Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

JAISTサマースクール2016「脳を知るための理論」講義02 Synaptic Learning rules

662 views

Published on

http://www.jaist.ac.jp/whatsnew/event/2016/07/07-3.html

Published in: Education
  • Login to see the comments

JAISTサマースクール2016「脳を知るための理論」講義02 Synaptic Learning rules

  1. 1. SS2016 Modern Neural Computation Lecture 2: Synaptic Learning Rules Hirokazu Tanaka School of Information Science Japan Institute of Science and Technology
  2. 2. Neurons communicate through synapses. In this lecture we will learn: • Basic anatomy and physiology of synapses • Rate coding and spike coding • Hebbian learning • Spike-timing-dependent plasticity • Reward-modulated plasticity
  3. 3. Synaptic plasticity underlies behavioral modification. Kandel (1979) Scientific American; Kandel (2001) Science
  4. 4. Synapses: electrical and chemical neurotransmission Figure 5.1, Neuroscience 3rd Edition
  5. 5. Long-term potentiation (LTP) of hippocampal synapses Figure 24.6, Neuroscience 3rd EditionFigure 24.5, Neuroscience 3rd Edition
  6. 6. Long-term potentiation (LTP) of hippocampal synapses Figures 24.7 & 24.8, Neuroscience 3rd Edition
  7. 7. Molecular mechanisms underlying hippocampal LTP. Figures 24.9 & 24.10, Neuroscience 3rd Edition
  8. 8. Long-term depression (LTD)
  9. 9. How does a neuron represent information? Panzari et al. (2010) Trends in Neurosciences
  10. 10. Rate coding: Number of Spikes matters. Rate coding hypothesis: a neuron represents information through its spike rate. Hartline (1940) Am J Physiol; Hartline (1969) Science Compound eye of horseshoe crab Recoding from optic nerve Firing patterns of cortical neurons are highly irregular, which are well approximated by a random Poisson process (Softky & Koch (1993) J Neurosci; Shadlen & Newsome (1994) Current Biology).
  11. 11. Temporal coding: Spike timing matters. Temporal coding hypothesis: a neuron represents information through its spike timings. Gollisch & Meister (2008) Science Johansson & Birznieks (2004) Nature Neurosci
  12. 12. Hebb’s postulate of activity dependent plasticity. "Let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased." Hebbian theory: a theory in neuroscience that proposes an explanation for the adaptation of neurons in the brain during the learning process. Donald O. Hebb (1904-1985) The Organization of Behavior (1949) Image source: Wikipedia, Donald O. Hebb
  13. 13. Synaptic plasticity: rate-coding model 1u 2u 3u 1w 2w 3w T v vτ =− + w u v ( ) ( ) T 1 T 1 n n u u w w = = u w   input ratesoutput rate synaptic strengths T v ≈ w u If we consider a time scale larger than τ, then
  14. 14. Hebbian plasticity in equation. vη∆ =w u 1 1 n n w u v w u η ∆        =       ∆      T η∆ =w uu w Hebbian learning with input vector u and output v Vector form: Or component form: If the membrane dynamics is fast compared to the timescale of synaptic plasticity, the output is approximated as: Then the Hebbian rule now reads: T .v = w u
  15. 15. This form of learning rule is unstable. T η∆ =w uu w T η η∆= =w uu w Cw Covariance matrix of random inputs T =C uu Wishart matrix If inputs u1, …, un are i.i.d., their covariance matrix is called the Wishart matrix (Wishart, 1936): All eigenvalues of a Wishart matrix are non-negative. Hebbian learning with single input u Hebbian learning with input ensemble Exercise 1
  16. 16. This form of learning rule is unstable. Eigenvalue decomposition 1, ,i i i i nλ= =Ce e  1 0nλ λ≥ ≥ ≥ η∆ =w Cw i i i a= ∑w e i i ia aηλ∆ = All eigenvalues of a Wishart matrix are non-negative. The eigenvectors form a basis for the n-dim space, and the weight vector w may be decomposed into the eigenvectors: Then, the Hebbian learning rule is reduced as: Therefore, ai grows exponentially, finally diverging to infinity.
  17. 17. Covariance matrix of input has non-negative eigenvalues. Covariance matrix of random inputs ( )T 2T T T 0≥= =x Cx x uu x u x 1 i i n i a e = = ∑x T , 1 , 2 1 , 1 T n n n i i j i j i j j i j j i i i j ia a a a aλ δ λ = = = = = =∑ ∑ ∑x Cx e Ce For any non-zero vector x: If the vector is decomposed in terms of eigenvectors, then, For any {ai} this quantity must be non-negative. Therefore, the eigenvalues {λi} must be non-negative, too.
  18. 18. Generalization of Hebbian learning. ( )( )v vη∆ = − −w u u Covariance learning BCM rule ( )Mv vη θ∆= −w u Bienenstock, Cooper & Munro (1982) J Neurosci Sejnowski (1977) Biophys J φ(v) v Synaptic weights change if pre-and post-activities are positively correlated. Synaptic plasticity depends linearly on pre- synaptic activities and nonlinearly on post- synaptic activity (thresholding). The thresholding value changes according to post-synaptic activity (homeostasis).
  19. 19. Generalization of Hebbian learning. BCM rule ( )Mv vη θ∆= −w u Bienenstock, Cooper & Munro (1982) J Neurosci φ(v) v Synaptic plasticity depends linearly on pre- synaptic activities and nonlinearly on post- synaptic activity (thresholding). The thresholding value changes according to post-synaptic activity (homeostasis). 2 EM vθ  =   ( )2 1v vη∆= −w u There is only one stable fixed point at v=1.
  20. 20. Weight normalization: additive or multiplicative. vη∆ =w uHebbian learning, , is inherently unstable. One way to avoid this instability (i.e., divergence) is to impose a constraint over the weight vector w. 1i i w =∑ Additive normalization Multiplicative normalization i i j j w w v w v n η η∆ = − ∑ ( ) ( ) ( ) ( ) ( ) 1 t t t t t + ∆ + = + ∆ w w w w w 1=w Oja (1982) Neural Networks
  21. 21. Oja learning rule as a principle component analyzer. Oja learning rule in discrete time Oja (1982) Neural Networks ( ) ( ) ( )2 1 v t v v v η η η η + + = = + − + + w u w w u w w u  ( ) ( ) ( ) ( ) ( ) ( )( )1t t v t t v t tη+ = + −w w u w ( ) d v v dt η= − w u w ( )Td dt η= − w Cw w Cww Oja learning rule in continuous time Oja learning rule in continuous time
  22. 22. Oja learning rule as a principle component analyzer. Oja (1982) Neural Networks ( )Td dt η= − w Cw w Cww i i i a= ∑w e 1, ,i i i i nλ= =Ce e  1 0nλ λ≥ ≥ ≥ 2 1 n i i i j j i j a a a aλ λ =   = −    ∑ 1 i i a b a ≡ ( )1i i ib bλ λ= − ( )1 const, 0 2, ,ia a i n∴ → → = Eigenvector decomposition
  23. 23. Modeling synapses: conductance-based model. ( )( ) ( )( )rest ex ex in inm dV V V g t E V g t E V dt τ = − + − + − LIF excitatory synapse inhibitory synapse Gerstner (2014) Neuronal Dynamics, Chapter 3 ( ) ( )syn syn syn f t t t f g t g e t t τ − − = Θ −∑ ( ) ( ) ( )rise fast slow syn syn 1 1 f f f t t t t t t f f g t g e ae a ae t tτ τ τ − − − − − −       = − + − Θ −        ∑ exponential with one decay time constant exponentials with one rise and two decay time constants
  24. 24. Modeling synapses: conductance-based model. ( )( ) ( )( )rest ex ex in inm dV V V g t E V g t E V dt τ = − + − + − LIF excitatory synapse inhibitory synapse Gerstner (2014) Neuronal Dynamics, Chapter 3 ( ) ( )syn syn syn f t t t f g t g e t t τ − − = Θ −∑ ( ) ( ) ( )rise fast slow syn syn 1 1 f f f t t t t t t f f g t g e ae a e t tτ τ τ − − − − − −       = − + − Θ −        ∑ exponential with one decay time constant exponentials with one rise and two decay time constants
  25. 25. Modeling synapses: conductance-based model. Gerstner (2014) Neuronal Dynamics, Chapter 3 ( ) ( )syn syn syn f t t t f g t g e t t τ − − = Θ −∑ ( ) ( ) ( )rise fast slow syn syn 1 1 f f f t t t t t t f f g t g e ae a e t tτ τ τ − − − − − −       = − + − Θ −        ∑ exponential with one decay time constant exponentials with one rise and two decay time constants excitatory inhibitory rise fast1 ms, 6 msτ τ≈ ≈ rise fast slow25 50 ms, 100 300 ms, 500 1000 msτ τ τ≈ − ≈ − ≈ − GABAA GABAB
  26. 26. Modeling synapses: conductance-based model. ( )( ) ( )( )rest ex ex in inm dV V V g t E V g t E V dt τ = − + − + − ex ex ex dg g dt τ = − ( ) ( )ex ex exg t g t g← + in in in dg g dt τ = − ( ) ( )in in ing t g t g← + LIF excitatory synapse inhibitory synapse Dynamics of conductance Synaptic plasticity: how the peak conductances of excitatory and inhibitory synapses is modified in an activity-dependent manner. Song et al. (2000) Nature Neurosci
  27. 27. Spike-timing dependent plasticity (STDP) Sjöström & Gerstner, Scholarpedia, 5(2):1362. doi:10.4249/scholarpedia.1362 pre-post: potentiation post-pre: depression
  28. 28. STDP in equations. Sjöström & Gerstner, Scholarpedia, 5(2):1362. doi:10.4249/scholarpedia.1362 ( ) :post spikes : pre spikes n f ij i j n f w W t t∆= −∑ ∑ ( ) exp for 0 exp for 0 tA t W t tA t τ τ + + − −   − >    =   − <    
  29. 29. Online implementation of STDP learning Sjöström & Gerstner, Scholarpedia, 5(2):1362. doi:10.4249/scholarpedia.1362 ( ) ( ) : presynaptic spike j f j j j f dx x a x t t dt τ δ+ +=− + −∑ ( ) ( ) : postsynaptic spike ni j i i n dy y a y t t dt τ δ− −=− + −∑ xj : presynaptic trace of neuron j “remembering when presynaptic neuron j spikes” yi : postsynaptic trace of neuron i “remembering when postsynaptic neuron i spikes” ( ) ( ) ( ) ( ) :postsynaptic : presynaptic spikes spikes ij n f ij j i ij i i n f dw A w x t t A w y t t dt δ δ+ −− − −∑ ∑
  30. 30. Weight dependence: hard and soft bounds. Sjöström & Gerstner, Scholarpedia, 5(2):1362. doi:10.4249/scholarpedia.1362 ( ) ( ) ( ) ( ) :postsynaptic : presynaptic spikes spikes ij ij ij n f j i i i n f dw x t t y t tA w A w dt δ δ+ −= − − −∑ ∑ Weight learning dynamics Hard bound rule (Linear) Soft bound rule ( ) ( ), :A w A w+ − determines the weight dependence of STDP learning rule. ( ) ( ) ( ) ( ) maxA w w w A w w η η + + − − =Θ − = Θ For biological reasons, the synaptic weights should be restricted to wmin < w < wmax . ( ) ( ) ( ) maxA w w w A w w η η + + − − = − = ( )A w+ ( )A w−
  31. 31. Temporal all-to-all versus nearest-neighbor spike interaction. Sjöström & Gerstner, Scholarpedia, 5(2):1362. doi:10.4249/scholarpedia.1362 ( ) ( ) ( ) ( ) : presynaptic : postsynaptic spike spike ,j f ni j j j i f n j i dx dy x t t y t t dt d a a y t xτ δ τ δ+ −+ −=− + − =− + −∑ ∑ Synaptic trace dynamics ( ):a x+ determines how much trace is incremented by spikes. ( ) 1a x+ = ( ) 1a x x+ = − All-to-all interaction Nearest-neighbor interaction All spikes contribute additively to the trace, and the trace is not upper-bounded. Only the nearest spike contributes to the trace and the trace is upper-bounded to 1.
  32. 32. Additive vs multiplicative STDP. van Rossum et al. (2000) J Neurosci. ( ) exp for 0 exp for 0 tA t W t tA t τ τ + + − −   − >    =   − <     ( ) exp for 0 exp for 0 tA t W t tA tW τ τ + + − −   − >    =   − <     Additive STDP Multiplicative STDP Potentiation and depression are independent of the weight value. Depression are weight dependent in a multiplicative way; a large synapse gets depressed more and a weak synapse less.
  33. 33. Triplet law: three-spike interaction pre pre1 2 pre1 1 1 1 pre2 2 2 2 post post1 2 post1 1 1 1 post2 2 2 2 if then 1. if then 1. if then 1. if then 1. dx dx x t t x x x t t x x dt dt dy dy y t t y y y t t y y dt dt τ τ τ τ =− = ← + =− = ← + =− = ← + =− = ← + ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )pre post 2 1 3 2 1 2 1 3 2 1w t A y t A x t y t t t A x t A y t x t t tε δ ε δ− − + +    ∆ =− + − − + + − −    post-pre LTD pre-post-pre LTD pre-post LTP post-pre-post LTP Pfister & Gerstner (2006) J Neurosci Dynamics of two presynaptic and two postsynaptic traces Pre-post-pre LTD and pre-post-pre LTP
  34. 34. STDP for inhibitory synapses Vogels et al. (2011) Science
  35. 35. Relation of STDP to other learning rules. • STDP and rate-based Hebbian learning rules Kempter, R., Gerstner, W., & Van Hemmen, J. L. (1999). Hebbian learning and spiking neurons. Physical Review E, 59(4), 4498. • STDP and Bienenstock-Cooper-Munro (BCM) rule Izhikevich, E. M., & Desai, N. S. (2003). Relating stdp to bcm. Neural computation, 15(7), 1511-1523. Pfister, J. P., & Gerstner, W. (2006). Triplets of spikes in a model of spike timing- dependent plasticity. The Journal of neuroscience, 26(38), 9673-9682. • STDP and temporal-difference learning rule Rao, R. P., & Sejnowski, T. J. (2001). Spike-timing-dependent Hebbian plasticity as temporal difference learning. Neural computation, 13(10), 2221-2237. Exercise 2
  36. 36. Functional consequence: reduced latency Sjöström & Gerstner, Scholarpedia, 5(2):1362. doi:10.4249/scholarpedia.1362 Song & Abbott (2000) Nature Neurosci potentiated depressed
  37. 37. Functional consequence: latent pattern detection Masquelier et al. (2008) PLoS One; (2009) Neural Comput
  38. 38. Functional consequence: latent pattern detection Masquelier et al. (2008) PLoS One; (2009) Neural Comput
  39. 39. Functional consequence: latent pattern detection Masquelier et al. (2008) PLoS One; (2009) Neural Comput ( ) ( ) ( ) j i j j t t t wv t t tη ε= −− + ∑ Spike response model (SRM): membrane potential in integral form. action potential synaptic potential presynaptic spikepostsynaptic spike Spike-timing-dependent plasticity: presynaptic spike tj and postsynaptic spike ti if if i j i j t j i j j t j t i j t A ew t w e t t A tw τ τ− + − + − − −   + →   − < > 
  40. 40. Functional consequence: latent pattern detection Masquelier et al. (2008) PLoS One; (2009) Neural Comput
  41. 41. Bird-song learning: LMAN provides exploratory noise. Vocal motor pathway (VMP) • HVC (High vocal center) • RA Anterior forebrain pathway (AFP) • Area X • DLM • LMAN Kao et al. (2005) Nature
  42. 42. HVC-RA synaptic plasticity modulated by reward. Fiete & Seung (2007) J Neurophysiol.
  43. 43. Tripartite synaptic plasticity ( ) LMAN LM V 0 N CA H ( ) ( ) ( ) (( )) i tij ij ii j dW R t e t dt s tG t t dt t s s tRη η  ′ −  ′− ′= ∫ Fiete & Seung (2007) J Neurophysiol. Exercise 3 This tripartite learning rule indeed leads to reward maximization.
  44. 44. Summary • Synaptic plasticity refers to activity-dependent change of a synaptic weight between neurons, underlying the physiological basis for learning and memory. • Hebbian learning: “Fire together, wire together.” • Synaptic plasticity may be formulated in terms of rate coding or spike-timing coding. • Synaptic plasticity is determined not only among two connected neurons but also is modulated by other factors (e.g., reward, homeostasis).
  45. 45. Exercises 1. Prove that all eigenvalues of a Wishart matrix are positive semidefinite. 2. Read the following paper: Kempter, R., Gerstner, W., & Van Hemmen, J. L. (1999). Hebbian learning and spiking neurons. Physical Review E, 59(4), 4498. From the additive STDP learning rule, derive the following rate-based Hebbian learning rule (fi and fj are pre- and post-synaptic activity, respectively): 3. Read the following paper: Fiete, I. R., & Seung, H. S. (2006). Gradient learning in spiking neural networks by dynamic perturbation of conductances. Physical review letters, 97(4), 048104. Prove that the learning rule (slide 46) can be derived as a consequence of reward maximization. ij i j jw f f fα β∆ = +
  46. 46. Exercises: Code Implementation of Song et al. (2000) ( ) ( ) ( ) ( )( ) ( ) ( )( )m rest ex ex inin dV t V V t g t E V t g t E V t dt τ = − + − + − ( ) ( ) ( ) ( )ex in e iex in nx , dg t dg t g t g t dt dt τ τ=− =− Membrane dynamics Conductance dynamics ( ) ( )ex ex ag t g t g→ + when a-th excitatory input arrives ( ) ( )ini inng t g t g+→ when any inhibitory input arrives Goal: Implement the STDP rule in Song, Miller & Abbott (2000).
  47. 47. Exercises: Code Implementation of Song et al. (2000) STDP for presynaptic firing: ( )maxmax ,0a a a a g M tg g P P A+ →    → + + STDP for postsynaptic firing: when a-th excitatory input arrives ( )max maxmin ,a a ag P tg g A g M M − →   → +  + when output neuron fires Synaptic traces: ( ) ( ) ( ) ( )+ , a a dM t M t dt dP t P t dt τ τ − = − = −
  48. 48. Exercises: Code Implementation of Song et al. (2000) %% parameter setting: % LIF-neuron parameters: taum = 20/1000; Vrest = -70; Eex = 0; Ein = -70; Vth = -54; Vreset = -60; % synapse parameters: Nex = 1000; Nin = 200; tauex = 5/1000; tauin = 5/1000; gmaxin = 0.05; gmaxex = 0.015; % STDP parameters: Ap = 0.005; An = Ap*1.05; taup = 20/1000; taun = 20/1000; %simulation parameters: dt = 0.1/1000; T = 200; t = 0:dt:T; % input firing rates: Fex = randi([10 30], 1, Nex); Fin = 10*ones(1,Nin); %% simulation: V = zeros(length(t), 1); M = zeros(length(t), 1); P = zeros(length(t), Nex); gex = zeros(length(t), 1); gin = zeros(length(t), 1); V(1) = Vreset; ga = zeros(length(t), Nex); ga(1,:) = gmaxex*ones(1,Nex); disp('Now simulating LIF neuron ...'); tic; for n=1:length(t)-1 % WRITE YOUR CODE HERE: end toc;

×