Artificial Neural Network Introduction

Artificial Neural Network : Introduction
Credits:
Soft Computing Applications, Dr. Debasis Samanta, IIT Kharagpur

RC Chakraborty, www.myreaders.info
“Principles of Soft Computing” by S.N. Sivanandam & SN Deepa
Neural Networks, Fuzzy Logic and Genetic Algorithms by S. Rajasekaran and GAV Pai
Neural networks- a comprehensive foundation by Simon S Haykin
Biological nervous system
Biological nervous system is the most important part of many

living things, in particular, human beings.
brain is the center of human nervous system.
In fact, any biological nervous system consists of a large number

of interconnected processing units called neurons.
Each neuron is approximately 10µm long and they can operate in

parallel.
Typically, a human brain consists of approximately 1011 neurons

communicating with each other with the help of electrical
impulses.
Brain: Center of the nervous system
Computation
Neuron: Basic unit of nervous system
Basic structure of biological neuron

Neuron and its working
Figure shows a schematic of a biological neuron. There are different
parts in it : dendrite, soma, axon and synapse.
Dendrite : Extends away from the cell body and is the main input
to the neuron. It receives information from thousands of other
neurons. Receive information
Soma : It is also called a cell body, and just like as a nucleus of

cell. Process information
Axon : A long cylindrical fibre. Carries information to other neuron
Synapse : It is a junction where axon makes contact with the
dendrites of neighboring dendrites.
Nucleus: is located within the soma, contains genetic
information, directs protein synthesis, and supplies the energy
and the resources the neuron needs to function.
There is a chemical in each neuron called neurotransmitter.
A signal (also called sense) is transmitted across neurons by this
chemical.
That is, all inputs from other neuron arrive to a neurons through
dendrites.
These signals are accumulated at the synapse of the neuron and
then serve as the output to be transmitted through the neuron.
An action may produce an electrical impulse, which usually lasts
for about a millisecond.
Note that this pulse generated due to an incoming signal and all
signal may not produce pulses in axon unless it crosses a
threshold value.
Also, note that an action signal in axon of a neuron is commutative
signals arrive at dendrites which summed up at soma.
Definition of Neural Networks (NN)
According to the DARPA Neural Network Study (1988, AFCEA
International Press, p. 60):
• ... a neural network is a system composed of many simple

processing elements operating in parallel whose function is
determined by network structure, connection strengths, and the
processing performed at computing elements or nodes.
According to Haykin (1994), p. 2:
A neural network is a massively parallel distributed processor that has

a natural propensity for storing experiential knowledge and making it
available for use. It resembles the brain in two respects:
• Knowledge is acquired by the network through a learning
process.
• Interneuron connection strengths known as synaptic weights
are used to store the knowledge.
Artificial neural network
In fact, the human brain is a highly complex structure viewed as a

massive, highly interconnected network of simple processing
elements called neurons.
Artificial neural networks (ANNs) or simply we refer it as neural

network (NNs), which are simplified models (i.e. imitations) of the
biological nervous system, and obviously, therefore, have been
motivated by the kind of computing performed by the human brain.
The behavior of a biolgical neural network can be captured by a

simple model called artificial neural network.
Analogy between BNN and ANN
Dendrite
Soma
Axon
Synapse
Hippocampus: region of the brain that is associated primarily with memory.

Analogy between BNN and ANN
x1
w1
x2 w2
w3
x3
wn
xn
x1
w1
x2 w2
w3
x3
…..
wn
xn
Summation Threshold unit output

input weight
unit
Here, x1, x2, · · · , xn are the n inputs to the artificial neuron.
w1, w2, · · · , wn are weights attached to the input links.

Note that, a biological neuron receives all inputs through the

dendrites, sums them and produces an output if the sum is
greater than a threshold value.
The input signals are passed on to the cell body through the
synapse, which may accelerate or retard an arriving signal.
It is this acceleration or retardation of the input signals that is

modeled by the weights.
An effective synapse, which transmits a stronger signal will have a

correspondingly larger weights while a weak synapse will have
smaller weights.
Thus, weights here are multiplicative factors of the inputs to

account for the strength of the synapse.
Hence, the total input say I received by the soma of the artificial
neuron is
To generate the final output y , the sum is passed to a filter

Ф
Called transfer function, which releases the output.
That is, y = Ф(I)
Summation Threshold unit output

input weight
unit
A very commonly known transfer function is the thresholding

function.
In this thresholding function, sum (i.e. I) is compared with a

threshold value θ.
If the value of I is greater than θ, then the output is 1 else it is 0

(this is just like a simple linear filter).
In other words,
where
Such a Φ is called step function(also known as Heaviside function).

Thresholding functions can be of two types: Binary (1, 0) or Bipolar [-1, 1].
output
+1.0
(I)
(I)
0 input -1.0
(a) Binary (Hard-limit) function (b) Bipolar Threshold function

Step vs Thresholding functions
Binary Step Binary Threshold
Bipolar Step Bipolar Threshold

Transformation functions
Hard-limit transfer function : The transformation we have just

discussed is called hard-limit transfer function. It is generally used
in perceptron neuron.
In other words,
Binary Threshold
Bipolar transfer function : The output of the transfer function is

lies in the range of −1.0 to +1.0. It is also known Quantizer
function and it defined as
Bipolar Threshold
Transformation functions
Signum transfer function
In Bipolar Threshold function When � = 0, it is same as ��(�).
1 if weighted sum (�) is positive.
0 if weighted sum (�) is zero.
-1 if weighted sum (�) is negative.

First use of the Transformation functions
The first mathematical model of a biological neuron was presented

by McCulloch and Pitts. (1943).
Hard limit activation function were used in the model.
McCulloch-Pitts model does not exhibit any learning but just serves
as a basic building block for significant work in NN research.
Other transformation functions
Piecewise Linear: a connected sequence of line segments.

This is also called saturating linear function and can have
either binary or bipolar.
1 �� > 1
∅(�) = � �� − 1 ≤ � ≤ 1
−1 �� <− 1
Piecewise Linear can have other forms.

Sigmoid transfer function : This function is a continuous

function that varies gradually between the asymptotic values 0
and 1 (called log-sigmoid) or -1 and +1 (called Tan-sigmoid)
function and is given by
Here, α is the coefficient of transfer function/ Slope parameter.

Unipolar Sigmoidal
Transfer functions in ANN
α=10
1
(I) α=0.3 1 α=10
α=1.0 α=0.5
(I) α=0.3
-1 1 -1 1
α=1.0 α=0.5
0
-1 -1
(a) Log-Sigmoid transfer function (b) Tan-Sigmoid transfer function

(Unipolar Sigmoidal) (Bipolar Sigmoidal)
Advantages of ANN
ANNs exhibits mapping capabilities, that is, they can map input
patterns to their associated output pattern.
The ANNs learn by examples. Thus, an ANN architecture can be

trained with known example of a problem before they are tested for
their inference capabilities on unknown instance of the problem. In
other words, they can identify new objects previously untrained.
The ANNs posses the capability to generalize. This is the power

to apply in application where exact mathematical model to
problem are not possible.
Advantages of ANN
The ANNs are robust system and fault tolerant. They can
therefore, recall full patterns from incomplete, partial or noisy
patterns.
The ANNS can process information in parallel, at high speed and

in a distributed manner. Thus a massively parallel distributed
processing system made up of highly interconnected (artificial)
neural computing elements having ability to learn and acquire
knowledge is possible.

Artificial Neural Network Introduction

Uploaded by

Copyright:

Available Formats

Artificial Neural Network Introduction

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Artificial Neural Network Introduction

Uploaded by

Copyright:

Available Formats

Artificial Neural Network : Introduction

Soft Computing Applications, Dr. Debasis Samanta, IIT Kharagpur

Biological nervous system is the most important part of many

brain is the center of human nervous system.

In fact, any biological nervous system consists of a large number

Each neuron is approximately 10µm long and they can operate in

Typically, a human brain consists of approximately 1011 neurons

Basic structure of biological neuron

Soma : It is also called a cell body, and just like as a nucleus of

• ... a neural network is a system composed of many simple

According to Haykin (1994), p. 2:

A neural network is a massively parallel distributed processor that has

In fact, the human brain is a highly complex structure viewed as a

Artificial neural networks (ANNs) or simply we refer it as neural

The behavior of a biolgical neural network can be captured by a

Hippocampus: region of the brain that is associated primarily with memory.

Summation Threshold unit output

Here, x1, x2, · · · , xn are the n inputs to the artificial neuron.

w1, w2, · · · , wn are weights attached to the input links.

Note that, a biological neuron receives all inputs through the

It is this acceleration or retardation of the input signals that is

An effective synapse, which transmits a stronger signal will have a

Thus, weights here are multiplicative factors of the inputs to

To generate the final output y , the sum is passed to a filter

Summation Threshold unit output

A very commonly known transfer function is the thresholding

In this thresholding function, sum (i.e. I) is compared with a

If the value of I is greater than θ, then the output is 1 else it is 0

Such a Φ is called step function(also known as Heaviside function).

(a) Binary (Hard-limit) function (b) Bipolar Threshold function

Binary Step Binary Threshold

Bipolar Step Bipolar Threshold

Hard-limit transfer function : The transformation we have just

Bipolar transfer function : The output of the transfer function is

Signum transfer function

In Bipolar Threshold function When � = 0, it is same as ���(�).

1 if weighted sum (�) is positive.

0 if weighted sum (�) is zero.

-1 if weighted sum (�) is negative.

The first mathematical model of a biological neuron was presented

Hard limit activation function were used in the model.

Piecewise Linear: a connected sequence of line segments.

Piecewise Linear can have other forms.

Sigmoid transfer function : This function is a continuous

Here, α is the coefficient of transfer function/ Slope parameter.

(I) α=0.3 1 α=10

(a) Log-Sigmoid transfer function (b) Tan-Sigmoid transfer function

The ANNs learn by examples. Thus, an ANN architecture can be

The ANNs posses the capability to generalize. This is the power

The ANNS can process information in parallel, at high speed and

You might also like

In Bipolar Threshold function When � = 0, it is same as ��(�).