Program: B.Tech, CSE, 6 Sem, 3 Year CS 601: Machine Learning Unit-4 Machine Learning: RNN in ML
Program: B.Tech, CSE, 6 Sem, 3 Year CS 601: Machine Learning Unit-4 Machine Learning: RNN in ML
Program: B.Tech, CSE, 6 Sem, 3 Year CS 601: Machine Learning Unit-4 Machine Learning: RNN in ML
Unit-4
Machine Learning: RNN in ML
Layer
Lecture No. 1
3
CS 601
Objective
4
CS 601
Recurrent neural network
Recurrent Neural Network (RNN) are a type of Neural Network
where the output from previous step are fed as input to the
current step.
In traditional neural networks, all the inputs and outputs are
independent of each other, but in cases like when it is required to
predict the next word of a sentence, the previous words are
required and hence there is a need to remember the previous
words.
Thus, RNN came into existence, which solved this issue with the
help of a Hidden Layer. The main and most important feature of
RNN is Hidden state, which remembers some information about
a sequence.
5
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Use cases of RNN
Recurrent neural network
7
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Recurrent neural network
parameters for each input as it performs the same task on all the
inputs or hidden layers to produce the output.
This reduces the complexity of parameters, unlike other neural
networks.
RNNs can take one or more input vectors and produce one or more
output vectors and the output(s) are influenced not just by weights
applied on inputs like a regular NN, but also by a “hidden” state
vector representing the context based on prior input(s)/output(s).
So, the same input could produce a different output depending on
previous inputs in the series.
8
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Recurrent neural network
Here, Ht is the new state, ht-1 is the previous state while xt is the
current input.
We now have a state of the previous input instead of the input
itself, because the input neuron would have applied the
transformations on our previous input.
So each successive input is called as a time step.
10
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Recurrent neural network
Taking the simplest form of a recurrent neural network, the
activation function is tanh, the weight at the recurrent neuron is
Whh and the weight at the input neuron is Wxh, we can write the
equation for the state at time t as –
11
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Training through RNN
1. A single time step of the input is supplied to the network i.e. xt is supplied to
the network
2. We then calculate its current state using a combination of the current input and
the previous state i.e. we calculate ht
3. The current ht becomes ht-1 for the next time step
4. We can go as many time steps as the problem demands and combine the
information from all the previous states
5. Once all the time steps are completed the final current state is used to calculate
the output yt
6. The output is then compared to the actual output and the error is generated
7. The error is then backpropagated to the network to update the weights and the
network is trained
12
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Types of RNN
One-to-one:
• This is also called Plain Neural networks. It deals with a
fixed size of the input to the fixed size of output, where
they are independent of previous information/output.
18
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Disadvantages of Recurrent Neural Network
1. Gradient vanishing and exploding problems.
2. Training an RNN is a very difficult task.
3. It cannot process very long sequences if using tanh or ReLU
as an activation function.
19
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Exercises
1. What Is the Difference Between a Feedforward Neural
Network and Recurrent Neural Network?
20
CS 601
Learning Outcomes
21
CS 601
Student Effective Learning Outcomes
22
CS 601
Text & Reference Books
Text Book:
1. Tom M. Mitchell, “Machine Learning”, McGraw Hill Education, First
edition, 2017
2. Ian Goodfellow and Yoshua Bengio and Aaron Courville, “Deep
Learning”, MIT Press, 2016
Reference Book:
1. Andreas Muller, "Introduction to Machine Learning with Python: A
Guide for Data Scientists", Shroff/O'Reilly; First edition (2016).
2. Aurelien Geon, “Hands-On Machine Learning with Scikit-Learn and
Tensorflow: Concepts, Tools, and Techniques to Build Intelligent
Systems”, Shroff/O'Reilly; First edition (2017).
23
CS 601
24
CS 601