SlideShare a Scribd company logo

Neural Network Design: Chapter 18 Grossberg Network

•Download as PPTX, PDF•
•
Jason Tsai
Jason Tsai

Lecture for Neural Networks study group held on January 11, 2020. Reference book: http://hagan.okstate.edu/nnd.html Video: https://youtu.be/H4NKgliTFUw Initiated by Taiwan AI Group (https://www.facebook.com/groups/Taiwan.AI.Group/permalink/2017771298545301/)

Technology
1 of 44
Chapter 18 Grossberg Network Jason Tsai (蔡志順) January 11, 2020 Mozilla Community Space Taipei
*Copyright Notice: Most materials from this presentation are taken from the book “Neural Network Design 2nd edition” authored by Martin T. Hagan, Howard B. Demuth, Mark Hudson Beale and Orlando De Jesús. The other quoted sources are mentioned in the respective slides. This presentation itself adopts Creative Commons license.
A Quote from this Chapter “Although the original inspiration for the field of artificial neural networks came from biology, at times we forget to look back to biology for new ideas. It will be the blending of biology, mathematics, psychology and other disciplines that will provide the maximum growth in our understanding of neural networks.” 「雖然類神經網路這一領域的最初靈感來自生物學， 但有時我們會忘了回顧生物學以啟迪新的想法。融合 生物學、數學、心理學和其他學科將使我們對神經網 路的理解得以最大的發展。」
Vision: Eyeball & Retina
Visual Pathway
Photograph of the Retina
Blind Spot *Figure adopted from https://bit.ly/39iUtNQ
Test for the Blind Spot Look at the blue circle on the left side of above figure with your right eye while covering your left eye and move your head back and forth perpendicularly to the circle to find the point at which the circle on the right will disappear from your field of vision.
Imperfections in Retinal Uptake
Compensatory Processing
Visual Illusions
Neon Color Spreading *Figure adopted from https://bit.ly/37xDyWg
Oriented Receptive Field *Figure adopted from https://bit.ly/2rPXGDF & https://bit.ly/2tmGg1X
Oriented Receptive Field (cont.)
Vision Normalization
Brightness Contrast
Lateral Inhibition *Figure adopted from https://bit.ly/2yaat37
Leaky Integrate-and-Fire *Figure adopted from Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski “Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition” Cambridge University Press. 2014. Page 11.
Leaky Integrator
*Solution to 1st order ODE *Credit to Schwartz Lyu
Leaky Integrator Response
Shunting Model Here p+, p-, b+, are b- are nonnegative values Refer to slide 28: Analysis of Normalization
Shunting Model Response
Grossberg Competitive Network
Layer 1
Characteristics of Layer 1 • The network is sensitive to relative intensities of the input pattern, rather than absolute intensities. • The output of Layer 1 is a normalized version of the input pattern. • The on-center/off-surround connection pattern and the nonlinear gain control of the shunting model produce the normalization effect. • The operation of Layer 1 explains the brightness constancy and brightness contrast characteristics of the human visual system.
Operation of Layer 1
Analysis of Normalization
Example of Layer 1 Response The response of the network maintains the relative intensities of the inputs, while limiting the total response.
Layer 2
Characteristics of Layer 2 • As in the Hamming and Kohonen networks, the inputs to Layer 2 are the inner products between the prototype patterns (rows of the weight matrix W2) and the output of Layer 1 (normalized input pattern). • The nonlinear feedback enables the network to store the output pattern (pattern remains after input is removed). • The on-center/off-surround connection pattern causes contrast enhancement (large inputs are maintained, while small inputs are attenuated).
Layer 2 Operation
Example of Layer 2 Response
Choice of Transfer Function
Sigmoid Transfer Function • A sigmoid function is faster-than-linear for small signals, approximately linear for intermediate signals and slower-than-linear for large signals. • When a sigmoid transfer function is used in Layer 2, the pattern is contrast enhanced; larger values are amplified, and smaller values are attenuated. • All initial neuron outputs that are less than a certain level decay to 0. • This merges the noise suppression of the faster- than-linear transfer functions with the perfect storage produced by linear transfer functions.
Adaptive Weights W2
Example of W2 Response
Equivalence to Discrete-time Rule
Relation to Kohonen Rule
Relation to Kohonen Rule (Cont.)
Three major differences: • The Grossberg network is a continuous-time network. • Layer 1 of the Grossberg network automatically normalizes the input vectors. • Layer 2 of the Grossberg can perform a “soft” competition, rather than the winner-take-all competition of the Kohonen network. This soft competition allows more than one neuron in Layer 2 to learn. This causes the Grossberg network to operate as a feature map. Grossberg vs. Kohonen Networks
Potential Issues • One key problem of the Grossberg network is the stability of learning. As more inputs are applied to the network, there is no guarantee that the weight matrix will eventually converge (form stable clusters / categories). [Refer to Chapter 19: Adaptive Resonance Theory, ART] • Another problem is the stability of the differential equations that implement Grossberg’s continuous- time competitive recurrent network. The output of a recurrent network could converge, oscillate, or even diverge. [Refer to Chapter 20: Stability]
Basic ART Architecture (Ch.19) Add-ons: Layer 2 to Layer 1 expectations The orienting subsystem (reset) Modified gain control Let’s move on…
References

Recommended

Neural network
Neural networkNeural network
Neural networkSilicon
 
Radial basis function network ppt bySheetal,Samreen and Dhanashri
Radial basis function network ppt bySheetal,Samreen and DhanashriRadial basis function network ppt bySheetal,Samreen and Dhanashri
Radial basis function network ppt bySheetal,Samreen and Dhanashrisheetal katkar
 
Neural Networks: Self-Organizing Maps (SOM)
Neural Networks: Self-Organizing Maps (SOM)Neural Networks: Self-Organizing Maps (SOM)
Neural Networks: Self-Organizing Maps (SOM)Mostafa G. M. Mostafa
 
Mc culloch pitts neuron
Mc culloch pitts neuronMc culloch pitts neuron
Mc culloch pitts neuronSiksha 'O' Anusandhan (Deemed to be University )
 
Hopfield Networks
Hopfield NetworksHopfield Networks
Hopfield NetworksKanchana Rani G
 
Artificial Neural Networks Lect3: Neural Network Learning rules
Artificial Neural Networks Lect3: Neural Network Learning rulesArtificial Neural Networks Lect3: Neural Network Learning rules
Artificial Neural Networks Lect3: Neural Network Learning rulesMohammed Bennamoun
 
Neural network
Neural networkNeural network
Neural networkRamesh Giri
 
Kohonen self organizing maps
Kohonen self organizing mapsKohonen self organizing maps
Kohonen self organizing mapsraphaelkiminya
 

Recommended

Neural network
Neural networkNeural network
Neural networkSilicon
 
Radial basis function network ppt bySheetal,Samreen and Dhanashri
Radial basis function network ppt bySheetal,Samreen and DhanashriRadial basis function network ppt bySheetal,Samreen and Dhanashri
Radial basis function network ppt bySheetal,Samreen and Dhanashrisheetal katkar
 
Neural Networks: Self-Organizing Maps (SOM)
Neural Networks: Self-Organizing Maps (SOM)Neural Networks: Self-Organizing Maps (SOM)
Neural Networks: Self-Organizing Maps (SOM)Mostafa G. M. Mostafa
 
Mc culloch pitts neuron
Mc culloch pitts neuronMc culloch pitts neuron
Mc culloch pitts neuronSiksha 'O' Anusandhan (Deemed to be University )
 
Hopfield Networks
Hopfield NetworksHopfield Networks
Hopfield NetworksKanchana Rani G
 
Artificial Neural Networks Lect3: Neural Network Learning rules
Artificial Neural Networks Lect3: Neural Network Learning rulesArtificial Neural Networks Lect3: Neural Network Learning rules
Artificial Neural Networks Lect3: Neural Network Learning rulesMohammed Bennamoun
 
Neural network
Neural networkNeural network
Neural networkRamesh Giri
 
Kohonen self organizing maps
Kohonen self organizing mapsKohonen self organizing maps
Kohonen self organizing mapsraphaelkiminya
 
Feedforward neural network
Feedforward neural networkFeedforward neural network
Feedforward neural networkSopheaktra YONG
 
02 Fundamental Concepts of ANN
02 Fundamental Concepts of ANN02 Fundamental Concepts of ANN
02 Fundamental Concepts of ANNTamer Ahmed Farrag, PhD
 
Back propagation
Back propagationBack propagation
Back propagationNagarajan
 
Artificial Neural Networks - ANN
Artificial Neural Networks - ANNArtificial Neural Networks - ANN
Artificial Neural Networks - ANNMohamed Talaat
 
Associative memory network
Associative memory networkAssociative memory network
Associative memory networkDr. C.V. Suresh Babu
 
HOPFIELD NETWORK
HOPFIELD NETWORKHOPFIELD NETWORK
HOPFIELD NETWORKankita pandey
 
Fuzzy inference systems
Fuzzy inference systemsFuzzy inference systems
Fuzzy inference systemsSiksha 'O' Anusandhan (Deemed to be University )
 
Max net
Max netMax net
Max netSandilya Sridhara
 
2.5 backpropagation
2.5 backpropagation2.5 backpropagation
2.5 backpropagationKrish_ver2
 
Multi Layer Perceptron & Back Propagation
Multi Layer Perceptron & Back PropagationMulti Layer Perceptron & Back Propagation
Multi Layer Perceptron & Back PropagationSung-ju Kim
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORKESCOM
 
Stochastic Gradient Decent (SGD).pptx
Stochastic Gradient Decent (SGD).pptxStochastic Gradient Decent (SGD).pptx
Stochastic Gradient Decent (SGD).pptxShubham Jaybhaye
 
Perceptron & Neural Networks
Perceptron & Neural NetworksPerceptron & Neural Networks
Perceptron & Neural NetworksNAGUR SHAREEF SHAIK
 
Feed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentFeed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentMuhammad Rasel
 
Ada boost
Ada boostAda boost
Ada boostHank (Tai-Chi) Wang
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural NetworksDatabricks
 
Perceptron and Sigmoid Neurons
Perceptron and Sigmoid NeuronsPerceptron and Sigmoid Neurons
Perceptron and Sigmoid NeuronsShajun Nisha
 
Multi-Layer Perceptrons
Multi-Layer PerceptronsMulti-Layer Perceptrons
Multi-Layer PerceptronsESCOM
 
Neural Networks: Support Vector machines
Neural Networks: Support Vector machinesNeural Networks: Support Vector machines
Neural Networks: Support Vector machinesMostafa G. M. Mostafa
 
Perceptron
PerceptronPerceptron
PerceptronNagarajan
 
Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksAdri Jovin
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithmsaciijournal
 

More Related Content

What's hot

Feedforward neural network
Feedforward neural networkFeedforward neural network
Feedforward neural networkSopheaktra YONG
 
Back propagation
Back propagationBack propagation
Back propagationNagarajan
 
Artificial Neural Networks - ANN
Artificial Neural Networks - ANNArtificial Neural Networks - ANN
Artificial Neural Networks - ANNMohamed Talaat
 
2.5 backpropagation
2.5 backpropagation2.5 backpropagation
2.5 backpropagationKrish_ver2
 
Multi Layer Perceptron & Back Propagation
Multi Layer Perceptron & Back PropagationMulti Layer Perceptron & Back Propagation
Multi Layer Perceptron & Back PropagationSung-ju Kim
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORKESCOM
 
Stochastic Gradient Decent (SGD).pptx
Stochastic Gradient Decent (SGD).pptxStochastic Gradient Decent (SGD).pptx
Stochastic Gradient Decent (SGD).pptxShubham Jaybhaye
 
Feed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentFeed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentMuhammad Rasel
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural NetworksDatabricks
 
Perceptron and Sigmoid Neurons
Perceptron and Sigmoid NeuronsPerceptron and Sigmoid Neurons
Perceptron and Sigmoid NeuronsShajun Nisha
 
Multi-Layer Perceptrons
Multi-Layer PerceptronsMulti-Layer Perceptrons
Multi-Layer PerceptronsESCOM
 
Neural Networks: Support Vector machines
Neural Networks: Support Vector machinesNeural Networks: Support Vector machines
Neural Networks: Support Vector machinesMostafa G. M. Mostafa
 

What's hot (20)

Feedforward neural network
Feedforward neural networkFeedforward neural network
Feedforward neural network
 
02 Fundamental Concepts of ANN
02 Fundamental Concepts of ANN02 Fundamental Concepts of ANN
02 Fundamental Concepts of ANN
 
Back propagation
Back propagationBack propagation
Back propagation
 
Artificial Neural Networks - ANN
Artificial Neural Networks - ANNArtificial Neural Networks - ANN
Artificial Neural Networks - ANN
 
Associative memory network
Associative memory networkAssociative memory network
Associative memory network
 
HOPFIELD NETWORK
HOPFIELD NETWORKHOPFIELD NETWORK
HOPFIELD NETWORK
 
Fuzzy inference systems
Fuzzy inference systemsFuzzy inference systems
Fuzzy inference systems
 
Max net
Max netMax net
Max net
 
2.5 backpropagation
2.5 backpropagation2.5 backpropagation
2.5 backpropagation
 
Multi Layer Perceptron & Back Propagation
Multi Layer Perceptron & Back PropagationMulti Layer Perceptron & Back Propagation
Multi Layer Perceptron & Back Propagation
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORK
 
Stochastic Gradient Decent (SGD).pptx
Stochastic Gradient Decent (SGD).pptxStochastic Gradient Decent (SGD).pptx
Stochastic Gradient Decent (SGD).pptx
 
Perceptron & Neural Networks
Perceptron & Neural NetworksPerceptron & Neural Networks
Perceptron & Neural Networks
 
Feed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentFeed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descent
 
Ada boost
Ada boostAda boost
Ada boost
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural Networks
 
Perceptron and Sigmoid Neurons
Perceptron and Sigmoid NeuronsPerceptron and Sigmoid Neurons
Perceptron and Sigmoid Neurons
 
Multi-Layer Perceptrons
Multi-Layer PerceptronsMulti-Layer Perceptrons
Multi-Layer Perceptrons
 
Neural Networks: Support Vector machines
Neural Networks: Support Vector machinesNeural Networks: Support Vector machines
Neural Networks: Support Vector machines
 
Perceptron
PerceptronPerceptron
Perceptron
 

Similar to Neural Network Design: Chapter 18 Grossberg Network

Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksAdri Jovin
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithmsaciijournal
 
Web spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsWeb spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsaciijournal
 
Autoencoders for image_classification
Autoencoders for image_classificationAutoencoders for image_classification
Autoencoders for image_classificationCenk Bircanoğlu
 
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Universitat Politècnica de Catalunya
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Gaurav Mittal
 
Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Dr.(Mrs).Gethsiyal Augasta
 
Topology ppt
Topology pptTopology ppt
Topology pptboocse11
 
Handwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksHandwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksIRJET Journal
 
DEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONDEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONIRJET Journal
 
Basic Learning Algorithms of ANN
Basic Learning Algorithms of ANNBasic Learning Algorithms of ANN
Basic Learning Algorithms of ANNwaseem khan
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORKESCOM
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural networkDEEPASHRI HK
 
M7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfM7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfArushiKansal3
 
Artificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningArtificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningSujit Pal
 

Similar to Neural Network Design: Chapter 18 Grossberg Network (20)

Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural Networks
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
 
D028018022
D028018022D028018022
D028018022
 
Web spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsWeb spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithms
 
Autoencoders for image_classification
Autoencoders for image_classificationAutoencoders for image_classification
Autoencoders for image_classification
 
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)
 
Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
Handwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksHandwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural Networks
 
DEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONDEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTION
 
Basic Learning Algorithms of ANN
Basic Learning Algorithms of ANNBasic Learning Algorithms of ANN
Basic Learning Algorithms of ANN
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORK
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural network
 
M7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfM7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdf
 
Artificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningArtificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep Learning
 

More from Jason Tsai

基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介Jason Tsai
 
Neural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksNeural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksJason Tsai
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Jason Tsai
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
Reinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceReinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceJason Tsai
 
Deep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyDeep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyJason Tsai
 
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsDeep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsJason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 

More from Jason Tsai (13)

基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介
 
Neural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksNeural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis Networks
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
Reinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceReinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 Neuroscience
 
Deep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyDeep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical Methodology
 
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsDeep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 

Recently uploaded

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdflior mazor
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 

Recently uploaded (20)

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 

Neural Network Design: Chapter 18 Grossberg Network

  • 1. Chapter 18 Grossberg Network Jason Tsai (蔡志順) January 11, 2020 Mozilla Community Space Taipei
  • 2. *Copyright Notice: Most materials from this presentation are taken from the book “Neural Network Design 2nd edition” authored by Martin T. Hagan, Howard B. Demuth, Mark Hudson Beale and Orlando De Jesús. The other quoted sources are mentioned in the respective slides. This presentation itself adopts Creative Commons license.
  • 3. A Quote from this Chapter “Although the original inspiration for the field of artificial neural networks came from biology, at times we forget to look back to biology for new ideas. It will be the blending of biology, mathematics, psychology and other disciplines that will provide the maximum growth in our understanding of neural networks.” 「雖然類神經網路這一領域的最初靈感來自生物學， 但有時我們會忘了回顧生物學以啟迪新的想法。融合 生物學、數學、心理學和其他學科將使我們對神經網 路的理解得以最大的發展。」
  • 7. Blind Spot *Figure adopted from https://bit.ly/39iUtNQ
  • 8. Test for the Blind Spot Look at the blue circle on the left side of above figure with your right eye while covering your left eye and move your head back and forth perpendicularly to the circle to find the point at which the circle on the right will disappear from your field of vision.
  • 12. Neon Color Spreading *Figure adopted from https://bit.ly/37xDyWg
  • 13. Oriented Receptive Field *Figure adopted from https://bit.ly/2rPXGDF & https://bit.ly/2tmGg1X
  • 17. Lateral Inhibition *Figure adopted from https://bit.ly/2yaat37
  • 18. Leaky Integrate-and-Fire *Figure adopted from Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski “Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition” Cambridge University Press. 2014. Page 11.
  • 20. *Solution to 1st order ODE *Credit to Schwartz Lyu
  • 22. Shunting Model Here p+, p-, b+, are b- are nonnegative values Refer to slide 28: Analysis of Normalization
  • 26. Characteristics of Layer 1 • The network is sensitive to relative intensities of the input pattern, rather than absolute intensities. • The output of Layer 1 is a normalized version of the input pattern. • The on-center/off-surround connection pattern and the nonlinear gain control of the shunting model produce the normalization effect. • The operation of Layer 1 explains the brightness constancy and brightness contrast characteristics of the human visual system.
  • 29. Example of Layer 1 Response The response of the network maintains the relative intensities of the inputs, while limiting the total response.
  • 31. Characteristics of Layer 2 • As in the Hamming and Kohonen networks, the inputs to Layer 2 are the inner products between the prototype patterns (rows of the weight matrix W2) and the output of Layer 1 (normalized input pattern). • The nonlinear feedback enables the network to store the output pattern (pattern remains after input is removed). • The on-center/off-surround connection pattern causes contrast enhancement (large inputs are maintained, while small inputs are attenuated).
  • 33. Example of Layer 2 Response
  • 34. Choice of Transfer Function
  • 35. Sigmoid Transfer Function • A sigmoid function is faster-than-linear for small signals, approximately linear for intermediate signals and slower-than-linear for large signals. • When a sigmoid transfer function is used in Layer 2, the pattern is contrast enhanced; larger values are amplified, and smaller values are attenuated. • All initial neuron outputs that are less than a certain level decay to 0. • This merges the noise suppression of the faster- than-linear transfer functions with the perfect storage produced by linear transfer functions.
  • 37. Example of W2 Response
  • 40. Relation to Kohonen Rule (Cont.)
  • 41. Three major differences: • The Grossberg network is a continuous-time network. • Layer 1 of the Grossberg network automatically normalizes the input vectors. • Layer 2 of the Grossberg can perform a “soft” competition, rather than the winner-take-all competition of the Kohonen network. This soft competition allows more than one neuron in Layer 2 to learn. This causes the Grossberg network to operate as a feature map. Grossberg vs. Kohonen Networks
  • 42. Potential Issues • One key problem of the Grossberg network is the stability of learning. As more inputs are applied to the network, there is no guarantee that the weight matrix will eventually converge (form stable clusters / categories). [Refer to Chapter 19: Adaptive Resonance Theory, ART] • Another problem is the stability of the differential equations that implement Grossberg’s continuous- time competitive recurrent network. The output of a recurrent network could converge, oscillate, or even diverge. [Refer to Chapter 20: Stability]
  • 43. Basic ART Architecture (Ch.19) Add-ons: Layer 2 to Layer 1 expectations The orienting subsystem (reset) Modified gain control Let’s move on…