Undergraduate course content: Introduction and a historical review Neural network concepts Basic models of ANN Linearly separable functions Non Linearly separable functions NN Learning techniques Associative networks Mapping networks Spatiotemporal Network Stochastic Networks

1. Neural Networks
Dr. Randa Elanwar
Lecture 1

2. Lecture Content
• Introduction and a historical review:
– History of neurocomputing.
– Overview of neurocomputing.
2Neural Networks Dr. Randa Elanwar

3. History of Neurocomputing
• Alexander Bain (1873) claimed that both thoughts and body
activity resulted from interactions among neurons within the
brain.
• For Bain, every activity led to the firing of a certain set of neurons.
When activities were repeated, the connections between those
neurons strengthened. According to his theory, this repetition was
what led to the formation of memory.
• The general scientific community at the time was doubting Bain’s
theory because it required what appeared to be an excessively
large number of neural connections within the brain.
3Neural Networks Dr. Randa Elanwar

4. History of Neurocomputing
• It is now apparent that the brain is exceedingly complex
and that the same brain “wiring” can handle multiple
problems and inputs.
• Neural network theory has served both to:
– better identify how the neurons in the brain function
and
– provide the basis for efforts to create artificial
intelligence.
4Neural Networks Dr. Randa Elanwar

5. Overview of Neurocomputing
• The brain is said to be composed of natural neural network, I.e., mass of
highly non-linear parallel inter-connected computational units called
neurons.
• Each neuron is connected to many other neurons
• Neurons transmit signals to each other
• Whether a signal is sent, depends on the strength of the bond between two
neurons
• Whether a signal is transmitted is an all-or-nothing event (the electrical
potential in the cell body of the neuron is thresholded: fired up/ activated)
5Neural Networks Dr. Randa Elanwar

6. Brain vs. Digital Computers
- Computers require hundreds of cycles to simulate
a firing of a neuron.
- The brain can fire all the neurons in a single step.
Parallelism
- Serial computers require billions of cycles to
perform some tasks but the brain takes less than
a second.
e.g. Face Recognition
6Neural Networks Dr. Randa Elanwar
Overview of Neurocomputing

7. 7Neural Networks Dr. Randa Elanwar
Overview of Neurocomputing

8. Neural Networks Dr. Randa Elanwar 8
Overview of Neurocomputing
What are Neural Networks
• Neural Networks (NNs) are networks of neurons, for example, as found in
real (i.e. biological) brains.
• Artificial Neurons are crude approximations of the neurons found in brains.
They may be physical devices, or purely mathematical constructs.
• Artificial Neural Networks (ANNs) are networks of Artificial Neurons, and
hence constitute crude approximations to parts of real brains. They may be
physical devices, or simulated on conventional computers.

9. Neural Networks Dr. Randa Elanwar 9
Overview of Neurocomputing
What are Neural Networks
• From a practical point of view, an ANN is just a parallel computational
system, consisting of many simple processing elements, connected together
in a specific way in order to perform a particular task, which is difficult to
traditional (serial) computers.
• One should never lose sight of how crude the approximations are, and how
over-simplified our ANNs are compared to real brains.

10. Comparison between brain verses ANN
Neural Networks Dr. Randa Elanwar 10
Brain ANN
Speed Few ms. Few nano sec. massive parallel
processing
Size and
complexity
1011 neurons & 1015
interconnections
Depends on designer
Storage
capacity
Stores information in its
interconnection (synapse)
No Loss of memory
Contiguous memory locations
loss of memory may happen
sometimes.
Tolerance Has fault tolerance Limited fault tolerance:
Information gets disrupted when
interconnections are disconnected
Control
mechanism
Complicated involves
chemicals in biological neuron
Simpler in ANN

11. Neural Networks Dr. Randa Elanwar 11
NON-LINEARITY
It can model non-linear systems
INPUT-OUTPUT MAPPING
It can derive a relationship between a set of
input & output responses
ADAPTIVITY
The ability to learn allows the network to adapt
to changes in the surrounding environment
EVIDENTIAL RESPONSE
It can provide a confidence level to a given
solution
Advantages of NN

12. Neural Networks Dr. Randa Elanwar 12
CONTEXTUAL INFORMATION
Knowledge is presented by the structure of the
network. Every neuron in the network is
potentially affected by the global activity of all
other neurons in the network. Consequently,
contextual information is dealt with naturally in
the network.
FAULT TOLERANCE
Distributed nature of the NN gives it fault
tolerant capabilities
NEUROBIOLOGY ANALOGY
Models the architecture of the brain
Advantages of NN

13. Who uses NN
13Neural Networks Dr. Randa Elanwar

14. Neuro products and application areas
14Neural Networks Dr. Randa Elanwar

15. Neural Networks Dr. Randa Elanwar 15
How does the nervous system works?
• The human nervous system can be broken down into three stages that may be
represented in block diagram form as:
– The receptors collect information from the environment – e.g. photons on the retina.
– The effectors generate interactions with the environment – e.g. activate muscles.
– The flow of information/activation is represented by arrows – feed forward and
feedback.
Overview of Neurocomputing

16. Overview of Neurocomputing
The structure of a biological neuron
Neural Networks Dr. Randa Elanwar 16
•A biological neuron has three types
of main components; dendrites,
soma (or cell body) and axon.
•Dendrites receives signals from
other neurons.
•The soma, sums the incoming
signals. When sufficient input is
received, the cell fires; that is it
transmit a signal over its axon to
other cells.

17. 17
Neural network: Definition
• Neural network: information processing paradigm inspired
by biological nervous systems, such as our brain
• Structure: large number of highly interconnected
processing elements (neurons) working together
• Like people, they learn from experience (by example)
Neural Networks Dr. Randa Elanwar

18. Artificial Neural Network: Definition
• The idea of ANN: NNs learn relationship between cause
and effect or organize large volumes of data into
orderly and informative patterns.
• Definition of ANN: “Data processing system consisting
of a large number of simple, highly interconnected
processing elements (artificial neurons) in an
architecture inspired by the structure of the cerebral
cortex of the brain”
(Tsoukalas & Uhrig, 1997).
18Neural Networks Dr. Randa Elanwar

19. Why Study Artificial Neural Networks?
• Artificial Neural Networks are powerful computational systems
consisting of many simple processing elements connected together to
perform tasks analogously to biological brains.
• They are massively parallel, which makes them efficient, robust, fault
tolerant and noise tolerant.
• They can learn from training data and generalize to new situations.
• They are useful for brain modeling and real world applications
involving pattern recognition, function approximation, prediction, …
19Neural Networks Dr. Randa Elanwar

20. What are Artificial Neural Networks Used for?
• Brain modeling
– Models of human development – help children with developmental problems
– Simulations of adult performance – aid our understanding of how the brain
works
– Neuropsychological models – suggest remedial actions for brain damaged
patients
• Real world applications
– Financial modeling – predicting stocks, shares, currency exchange rates
– Other time series prediction – climate, weather, airline marketing tactician
– Computer games – intelligent agents, backgammon, first person shooters
– Control systems – autonomous adaptable robots, microwave controllers
– Pattern recognition – speech recognition, hand-writing recognition, sonar
signals
– Data analysis – data compression, data mining
– Noise reduction – function approximation, ECG noise reduction
– Bioinformatics – protein secondary structure, DNA sequencing
20Neural Networks Dr. Randa Elanwar