TRIBHUVAN UNIVERSITY

LALITPUR ENGINEERING COLLEGE

MID TERM PROJECT REPORT

ON
“ AMERICAN SIGN LANGUAGE DETECTION USING CNN”

SUBMITTED BY
Amrit Sapkota (076 BCT 05)
Asmit Oli (076 BCT 43)
Nischal Maharjan (076 BCT 20)
Sakshyam Aryal (076 BCT 29)

SUBMITTED TO:
DEPARTMENT OF COMPUTER ENGINEERING
LALITPUR ENGINEERING COLLEGE
LALITPUR, NEPAL

SUPERVISED BY
Er. Hemant Joshi

December, 2023
 TRIBHUVAN UNIVERSITY
INSTITUTE OF ENGINEERING
LALITPUR ENGINEERING COLLEGE
DEPARTMENT OF COMPUTER ENGINEERING

MID TERM PROJECT REPORT ON:

”AMERICAN SIGN LANGUAGE DETECTION USING CNN ”
IN PARTIAL FULFILLMENT FOR THE
AWARD OF
BACHELOR’S DEGREE IN COMPUTER ENGINEERING

SUBMITTED BY
Amrit Sapkota (076 BCT 05)
Asmit Oli (076 BCT 43)
Nischal Maharjan (076 BCT 20)
Sakshyan Aryal (076 BCT 29)

SUPERVISED BY
Er. Hemant Joshi

December, 2023
ACKNOWLEDGEMENT

First and foremost, we would like to thank our supervisor, Er. Hemant Joshi sir,
who guided us in doing this project. He provided us with invaluable advice and
helped us in difficult stages. His motivations helped tremendously to the successful
completion of the project. We are really grateful to our project coordinator, Er.
Bibat Thokar, for advising us and introducing the project to us in an easy-to-
understand way which has helped us to complete our project easily and effectively
on time. We would like to express our special thanks of gratitude to IOE as well
as our principal, Dr.Surendra Tamrakar, who gave us the golden opportunity to do
this wonderful project on the topic of sign language Detection, which also helped us
in a lot of research and we came to know about so many new things. We are really
thankful to them. Besides, we would like to thank all the teachers who helped us
by advising us and providing the equipment we needed. We are overwhelmed in
all humbleness and gratefulness to acknowledge our depth to all those who have
helped us to put these ideas, well above the level of simplicity and into something
concrete. Also, we would like to thank our family and friends for their support.
Without their support we wouldn’t have succeeded in completing this project. Last
but not the least, we would like to thank everyone who helped and motivate us to
work on this project.

Sincerely,
Amrit Sapkota (076 BCT 05)
Asmit Oli (076 BCT 43)
Nischal Maharjan (076 BCT 20)
Sakshyan Aryal (076 BCT 29)

i
ABSTRACT

There is an undeniable communication problem between the Deaf community and

the hearing majority. It becomes hard for deaf people to communicate because
many people don’t understand sign language. With the use of innovation, in sign
language recognition, we tried to teardown this communication barrier. In this
proposal, it is shown how using Artificial Intelligence can play a key role to provide
the solution. Using the dataset, through the front camera of the laptop, translation
of sign language to text format can be seen on the screen in real-time i.e. the input
is in video format whereas the output is in text format. Extraction of complex head
and hand movements along with their constantly changing shapes for recognition
of sign language is considered a difficult problem in computer vision. Mediapipe
provides necessary key points or landmarks of hand, face, and pose. The model is
then trained using a Convolutional neural network(CNN). The trained model is
used to recognize sign language.

Keywords: Convolution Neural Network(CNN), Recurrent Neural Network, Deep

Learning, Gesture Recognition, Sign Language Recognition.

ii
TABLE OF CONTENTS

ACKNOWLEDGEMENT i
ABSTRACT ii
TABLE OF CONTENTS iii
LIST OF FIGURES v
LIST OF ABBREVIATIONS vi
1 INTRODUCTION 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5 System Requirement . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5.1 Functional Requirements . . . . . . . . . . . . . . . . . . . 2
1.5.2 Non-functional Requirement . . . . . . . . . . . . . . . . . . 3
2 LITERATURE REVIEW 4
3 BLOCK DIAGRAM 6
3.1 System Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2 Use Case Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3 Level 0 DFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4 Level 1 DFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.5 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4 METHODOLOGY 11
4.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2.1 Video Acquisition . . . . . . . . . . . . . . . . . . . . . . . . 12
4.2.2 Video Segmentation . . . . . . . . . . . . . . . . . . . . . . . 12
4.2.3 Frame Extraction . . . . . . . . . . . . . . . . . . . . . . . . 12
4.2.4 Preprocessing Techniques . . . . . . . . . . . . . . . . . . . 12
4.2.5 Hand Segmentation . . . . . . . . . . . . . . . . . . . . . . . 13
4.2.6 Data Augmentation . . . . . . . . . . . . . . . . . . . . . . . 13

iii
 4.3 Convolution Neural Network . . . . . . . . . . . . . . . . . . . . . . 13
4.4 Loss Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5 IMPLEMENTATION PLAN 16
5.1 Gantt Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
6 REQUIREMENT ANALYSIS 17
6.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . 17
6.2 Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 17
6.3 User Requirement Definition . . . . . . . . . . . . . . . . . . . . . . 18
7 RESULT AND ANALYSIS 19
7.1 Quantitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 20
7.2 Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 22
7.3 Comparision of CNN and LSTM . . . . . . . . . . . . . . . . . . . . 23
8 EPILOGUE 24
8.1 Task Completed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
8.2 Remaining Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
REFERENCES 26

iv
LIST OF FIGURES

3.1 System Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.2 Use Case Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3 Level 0 DFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4 Level 1 DFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.5 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.2 CNN Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.1 Hand Landmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

7.1 Epoch Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

7.2 Epoch Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
7.3 Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
7.4 Output from Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
7.5 Comparision between LSTM and CNN . . . . . . . . . . . . . . . . 23

v
LIST OF ABBREVIATIONS

ConvNets Convolutional Networks

CNN Convolution Neural Network
CPU Central Processing Unit
DFD Data Flow Diagram
DTW Dynamic Time Warping
Et al. And Others
FPS Frames Per Second
GPU Graphic Processing Unit
LSTM One Dimensions
ReLu rectified linear unit
RGB Red Green Blue
1D One Dimensions
SVM Support Vector Machine
PCA Principle Component Analysis

vi
CHAPTER 1

INTRODUCTION

1.1 Background

With the rapid growth of technology around us, Machine Learning and Artificial
Intelligence has been used in various sectors to support mankind including gesture,
object, face detection etc. With the help of Deep Learning, a machine imitates
the way humans gain certain types of knowledge. Using Artificial Neural Network,
simulation of human brain is done and using Convolution layers, extraction of
selected important part from an image to make computation easy. ”Sign Language
Detection”, the name itself specifies the gist of the project. Sign language recog-
nition has been a major problem between mute disabilities people in community.
People does not understand sign language and also it is difficult for them to learn
those sign language. A part from the scoring grades form this minor project, the
core idea is to make communication easy for deaf people. We set the bar of the
project such that it would be beneficial to society as well. The main reason for us
to choose this project is to aid people using Artificial Intelligence.[? ]

1.2 Problem Definition

Though there is a lot of research going on regarding sign language recognition,

there is very little implementation in practical life. As per the team research, upto
that extent, we came across this problem though there are many Sign language
recognition software and hardware out there but to some extent we felt that
people who don’t understand sign language or can’t read may have the problem
in communication also recognition using gloves is practically not portable and
costly as well. This made us think to use image processing and deep learning for
sign language recognition and provide the output as voice such that everyone can
understand.[? ]

1
1.3 Scope

The field of sign language recognition includes the development and application of
techniques for recognizing and interpreting sign language gestures. This involves
using computer vision and machine learning techniques to analyze video input and
identify gestures of sign language users. Sign language recognition has a wide range
of potential applications, including communication aids for deaf people, automatic
translation of sign language into spoken or written language, and an interactive
platform for learning sign language. The scope also extends to improving the
accuracy and efficiency of sign language recognition systems through advances in
algorithms, sensor technology and data collection. Additionally, this scope also
includes addressing challenges related to sign language diversity, gestural variation,
lighting conditions, and the need for robust real-time performance in a variety of
environments. [? ]

1.4 Objectives

The following are the objectives for sign language detection.

• To design and implement a system that can understand the sign language of
Hearing-impaired people.

• To train the model with a variety of datasets using MediaPipe and CNN,
and provide the output in real-time.

• To recognize sign language and provide the output as voice or text.

1.5 System Requirement

The following is the desired functionality of the new system. The proposed project
would cover.

1.5.1 Functional Requirements

• Real-time Output.

• Accurate detection of gestures.

2
 • Data sets comment.

1.5.2 Non-functional Requirement

• Performance Requirement.

• Design Constraints.

• Reliability.

• Usability.

• Maintainability.

3
CHAPTER 2

LITERATURE REVIEW

A survey conducted in 2016 by Brandon Garcia Stanford University Stanford, CA

in their Real-time American Sign Language Recognition with Convolutional Neural
Networks used CNN algorithm and predicted the result with accuracy 95.72% [1].

Rahib H.Abiyev et al. in 2019 in their Sign Language Translation Using Deep
Convolutional Neural Networks paper used CNN and Finger-spelling dataset and
predicted the result with accuracy 94.91% [2].

A survey conducted in 2019 by IEEE in their User-Independent American Sign

Language Alphabet Recognition Based on Depth Image and PCANet Features
paper used Principal Component Analysis Network (PCANet) and predicted the
result with accuracy 88% [? ].

Md. Moklesur Rahman et al. in 2019 in their A New Benchmark on American

Sign Language Recognition using Convolutional Neural Network paper used CNN
and predicted the result with accuracy 95.9% [? ].

Jungpil Shin et al. in 2020 in their American Sign Language Alphabet Recognition
by Extracting Feature from Hand Pose Estimation paper used SVM and predicted
the result with accuracy 87% [? ].

Yulius Obi in 2023 in their Sign language recognition system for communicating to
people with disabilities paper used CNN and predicted the result with accuracy
95.1% [? ].

Jyotishman Bora in 2023 in their Real-time Assamese Sign Language Recognition

using MediaPipe and Deep Learning paper used Mediapipe, Microsoft Kinect
sensor and predicted the result with accuracy 96.21% [? ].

4
Table 2.1: Summary of Related Works on ASL Recognition

S.N. Related Works Results Tools Used

Real-time American
Sign Language Recognition
1 accuracy 95.72% CNN
with Convolutional Neural
Networks (2016)
Sign Language Translation
CNN,
Using
2 accuracy 94.91% Fingerspelling
Deep Convolutional Neural
dataset
Networks Rahib H (2019)
User-Independent American
Sign Language Alphabet Principal
Recognition Based on Depth Component
3 accuracy 88%
Image and PCA Net Analysis Network
Features Walah Aly (PCANet)
and Saleh Aly (2019)
A New Benchmark on
American Sign Language
Recognition using
4 accuracy 95.9% CNN
Convolutional Neural
Network.
Md. Rahman (2019)
American Sign Language
Alphabet Recognition by
5 Extracting Features from accuracy 87% SVM
Hand Pose Estimation.
(2020)
American sign language accuracy
recognition and training 93.36%(LSTM),
6 LSTM, SVM, RNN
method with recurrent 94.23%(SVM) ,
neural network lee (2021) 95.03%(RNN)
Real-time recognition of
American sign language
7 using long-short-term accuracy 93.81% LSTM
memory neural network and
hand detection (2021)
Sign language recognition
system for communicating
8 accuracy 95.1% CNN
to people with disabilities.
(2023)
Real-time Assamese Sign
Mediapipe,
Language Recognition using
9 accuracy 96.21% Microsoft Kinect
MediaPipe and Deep
sensor
Learning. (2023)

5
CHAPTER 3

BLOCK DIAGRAM

3.1 System Block Diagram

Figure 3.1: System Block Diagram

The overall workflow of the system is shown in the above block diagram. Data-set
are like the memory of the system. Each and every detection that we view in real
time are the results of the data-set. Data-sets are captured in real time from the
front camera of the laptop . Using media pipe live perception of simultaneous
human pose, face landmarks, and hand tracking in real-time various modern life
applications including sign language detection can be enabled. With the help of
the landmarks or let’s say key points of features (face, pose, and hands) we get
from the media pipe we train our model. All the data that we collected from the
data-sets and from deep learning models are considered as training data. These
data are provided to the system such that the system can detect the sign language
in real-time. Input to this system is real-time or say live video using the front
camera of the laptop. As the real-time input i.e. sign language is provided using
the front camera of laptop, simultaneously live output can be seen on the screen in
text format. It acts as an interface for the Sign Language System providing an
environment for input data to get processed and provide the output.

6
3.2 Use Case Diagram

Figure 3.2: Use Case Diagram

7
3.3 Level 0 DFD

Figure 3.3: Level 0 DFD

8
3.4 Level 1 DFD

Figure 3.4: Level 1 DFD

9
3.5 Activity Diagram

Figure 3.5: Activity Diagram

10
CHAPTER 4

METHODOLOGY

4.1 Data Collection

In this project, we have collected sign language data of around 20,000 and made
10 classes. These classes are then made labels and the predictions are made from
these labels. High-quality video recording tools, including cameras and lighting
setups that allow for good viewing of hand motions, will be used to collect the data
We have used the Media-pipe library to extract key points from the images, which
are stored as data. The sample of data collection is shown below.

Figure 4.1: Data Collection

4.2 Data Preprocessing

Data preprocessing is a process of preparing the raw data and making it suitable
for a machine-learning model. It is the first and crucial step in creating a machine-
learning model. Real-world data generally contains noises and missing values and
may be unusable, which cannot be directly used for machine, learning models.
Data preprocessing is a required task for cleaning the data and making it suitable
for a machine-learning model, which also increases the accuracy and efficiency of a

11
machine-learning model. In our project, we have used a media-pipe library that
does preprocessing for our data.

4.2.1 Video Acquisition

The video data for sign language detection will be captured using a high-definition
camera with a decent resolution. The camera will be positioned to capture the
frontal view of the signer’s upper body, focusing on the hand region.

4.2.2 Video Segmentation

Then the acquired video data will be segmented into individual sign language
gestures. We will employ an automatic gesture detection algorithm based on motion
and hand region analysis. This algorithm will then detect significant changes in
motion and will use hand-tracking techniques to separate consecutive gestures from
video sequences.

4.2.3 Frame Extraction

From the segmented video data, frames will be extracted at a rate of one frame
per second to capture key moments of each gesture. A sample set of frames will
thus be guaranteed for additional study.

4.2.4 Preprocessing Techniques

Resizing and Cropping

To maintain consistency in the input data, the extracted frames will be resized to a
small resolution for example 680x480 pixels. Additionally, we will crop each frame
to focus on the hand region, ensuring that irrelevant background information is
eliminated.
Color Conversion
We will convert the frames from RGB to grayscale to reduce the computational
complexity and focus solely on hand shape and motion.
Noise Reduction
Gaussian smoothing with a kernel size of 3x3 will be applied to reduce noise in
the grayscale frames. This will help to enhance the clarity of hand contours and

12
minimize the impact of minor variations caused by lighting conditions.
Contrast Enhancements
Histogram equalization will be applied to the grayscale frames to improve the
visibility of hand features. This will enhance the contrast and increase the overall
dynamic range of pixel intensities.
Normalization
We will use min-max scaling to translate the intensity values from the [0, 255]
range to [0, 1], standardizing the pixel values across frames. By ensuring that the
input data has consistent ranges, this normalization step will help in convergence
during model training.

4.2.5 Hand Segmentation

We will use a hand segmentation technique based on color and region analysis be-
cause hand movements are important in sign language. To separate the hands from
the background and other unimportant items, this technique will use background
subtraction and skin color modeling.

4.2.6 Data Augmentation

We will use data augmentation techniques to broaden the variety and amount
of the training dataset. These will consist of randomizing the frames’ cropping,
rotation, translation, and flipping. The model’s ability to recognize sign gestures
in a variety of situations will be strengthened with the aid of data augmentation.

4.3 Convolution Neural Network

The CNN layer is the most significant; it builds a convolved feature map by applying
a filter to an array of picture pixels. I developed a CNN with three layers, each layer
using convolution, ReLU, and pooling. Because CNN does not handle rotation and
scaling by itself, a data augmentation approach was used. A few samples have been
rotated, enlarged, shrunk, thickened, and thinned manually. Convolution filters
are applied to the input using 1D Convolutions to extract the most significant
characteristics. The kernel glides in one dimension in 1D convolution, which
exactly suits the spatial properties. Convolution sparsity, when used with pooling

13
 Figure 4.2: CNN Architecture

for location invariant feature detection and parameter sharing, lowers overfitting.
ReLU layer is a layer where data travels through each layer of the network, the
ReLU layer functions as an activation function, ensuring non-linearity. Without
ReLU, the dimensionality that is desired would be lost. It introduces non-linearity,
accelerates training, and reduces computation time. Pooling layer is a layer that
gradually decreases the dimension of the feature and variation of the represented
data. Decreases dimensions and computation, speeds up processing by reducing
the number of parameters that the network must compute, reduces overfitting by
reducing the number of parameters, and makes the model more tolerant of changes
and distortions. Pooling strategies include max pooling, min pooling, and average
pooling; I tried max pooling. The maximum input of a convolved feature is used in
max pooling. Flatten is used to transform the data into a one-dimensional array
for input to the next layer. Dense the weights are multiplied by a matrix-vector
multiplication of the input tensors, followed by an activation function. Apart from
an activation function, the essential argument that we define here is units, which is
an integer that we use to select the output size. Dropout layer is a regularisation
approach that eliminates neurons from layers at random, along with their input
and output connections. As a consequence, generality is improved, and overfitting
is avoided.

14
4.4 Loss Function

Categorical cross-entropy is a widely used loss function in machine learning, par-

ticularly in the context of multi-class classification tasks such as American Sign
Language (ASL) detection. Specifically tailored for scenarios where instances be-
long to one of several mutually exclusive classes, categorical cross-entropy measures
the dissimilarity between the predicted probability distribution of classes and the
true distribution. In the realm of ASL detection, where accurate classification of
various sign gestures is crucial, this loss function plays a pivotal role in guiding
the training process. By penalizing deviations from the actual class probabilities,
categorical cross-entropy effectively steers the model towards learning to make
more precise predictions. Its implementation ensures that the model is trained
to discern subtle differences among ASL gestures, ultimately contributing to en-
hanced accuracy and proficiency in sign language recognition. The documentation
should underscore the significance of categorical cross-entropy in the ASL detection
pipeline, elucidating its role in optimizing the model’s ability to interpret and
classify a diverse range of sign language expressions accurately.

N M
1 XX
Categorical Cross-Entropy = − yij log(pij ) (4.1)
N i=1 j=1

15
CHAPTER 5

IMPLEMENTATION PLAN

5.1 Gantt Chart

Table 5.1: Time Schedule for the Project Work

Weeks
Task 1 2 3 4 5 6 7 8 9 10 11 12
Literature Review
Proposal Defense
Implementation
Mid-term Defense
Testing and Debugging
Validation and Analysis
Final Submission of Project
Documentation
Coordination with Supervi-
sor

16
CHAPTER 6

REQUIREMENT ANALYSIS

6.1 Hardware Requirements

The hardware required for the projects are:

• CPU

• GPU

• Storage

6.2 Software Requirements

The software required for the projects is :

Python
Python is a high level language which is used for general purpose programming. It
was developed by Guido van Rossum. The first release of Python was in the year
1991 as Python 0.9.0. Programming paradigms such as structured, object oriented,
and functional programming are supported in Python.

TensorFlow
TensorFlow is a free open source library that can be used in the field of machine
learning and artificial intelligence. Including many other tasks, it can be used for
training purposes in deep learning.

Mediapipe
MediaPipe offers cross-platform, customizable machine learning solutions for live
and streaming media i.e. real time videos. Its features are End to end acceleration,
Build once deploy anywhere, ready to use solution, and Free and open source.

17
 Figure 6.1: Hand Landmarks

6.3 User Requirement Definition

The user requirement for this system is to make the system fast, feasible, less prone
to error, save time and improve the communication gap between normal people
and deaf people

• The system can translate sign language into text.

• The system should have a user-friendly interface.

18
CHAPTER 7

RESULT AND ANALYSIS

CNN model has been trained in 8 epochs with an accuracy of 97.52. We have used
the following parameters for training our model.

Table 7.1: Model Parameters

S.N. Parameter Used Value

1 Number of Convolutional Layers 3 (32,64,128)
2 Activation Functions ReLU
3 Learning Rate 0.001
4 Optimizer Adam
5 Batch Size 32
6 Epochs 13
7 Input Array Size (63,1)
8 Loss Function categorical crossentropy

After training our model we have visualized the accuracy and the loss curve.We
have found that the accuracy at epoch 1 was 0.7227 and validation accuracy was
0.9026, 0.9086 accuracy and 0.9407 validation accuracy at epoch 2 and goes on
increasing as shown in figure 7.1. Also we have found that the loss at epoch 1 was
0.9629 and validation loss was 0.2303, 0.2372 loss and 0.1435 validation loss at
epoch 2 and goes on decreasing as shown in figure 7.2.

Figure 7.1: Epoch Accuracy

19
 Figure 7.2: Epoch Loss

7.1 Quantitative Analysis

In order to evaluate the effectiveness of the system being proposed, we have

measured its performance using various metrics, including Accuracy, Precision,

Recall, F1-Score, and Error Rate. Accuracy refers to how closely the measurements

of the system align with a particular value, and is expressed as follows:

TP + TN
Accuracy = (7.1)
TP + FN + FP + TN

Precision is a metric that measures the accuracy of positive predictions

made by a system. It can be obtained by dividing true positives by
the sum of true positives and false positives.

TP
P recision = (7.2)
TP + FP

In machine learning, recall, also referred to as sensitivity or true

positive rate, represents the likelihood that the model accurately
recognizes the detected anomaly.

TP
Recall = (7.3)
TP + FN

The F1-Score is a metric that combines precision and recall using

20
their harmonic mean. It provides a single value for comparison, with
higher values indicating better performance.

2 ∗ P recision ∗ Recall
F 1 − Score = (7.4)
P recision + Recall

21
 Figure 7.3: Confusion Matrix

The above photo is the sign of the ’Hello’ word. The shown hand’s
landmarks are taken in NumPy array form, and then it is fed into our
deep learning model. Our model classified the gesture of our hand.
Hence it outputs a class label corresponding to the gesture.

7.2 Qualitative Analysis

Output from our model is shown below where ”Hello, no, I Love You
”, This kind of hand gestures input has been provided and output
can be seen on screen.

22
 Figure 7.4: Output from Model

7.3 Comparision of CNN and LSTM

Figure 7.5: Comparision between LSTM and CNN

23
CHAPTER 8

EPILOGUE

8.1 Task Completed

A significant milestone was achieved through the development and

training of both Long Short-Term Memory (LSTM) and Convolu-
tional Neural Network (CNN) models. The LSTM model, renowned
for its proficiency in sequence prediction tasks, underwent meticulous
training to grasp intricate patterns within the data. Simultaneously,
the CNN model, tailored for image-related tasks, underwent a rigor-
ous training regimen to enhance its feature extraction capabilities.
Model evaluation played a pivotal role in assessing the efficancy of
these trained models. Through a meticulous process of validation
and testing, their performance metrics were scrutinized to ensure
their aptitude for real-world applications.

To further elevate model efficiency, a multifaceted approach was im-

plemented. Firstly, the dataset was substantially expanded, exposing
the models to a more diverse range of examples, thereby augmenting
their ability to generalize. This step proved crucial in enhancing the
models’ robustness across a spectrum of scenarios. Additionally, the
number of units in both dense layers of the models was increased,
empowering them to capture more intricate relationships in the data.
This strategic augmentation of model complexity contributed to a
more nuanced understanding of the underlying patterns.

24
Furthermore, a deliberate increase in kernel size and the number of
filters in the CNN model was undertaken. This adjustment facilitated
a broader scope of feature extraction, enabling the model to discern
more complex spatial hierarchies within the input data. As a result,
the model’s capacity for recognizing subtle patterns in images was
greatly amplified. Collectively, these refinements, encompassing
dataset expansion, augmentation of model architecture, and fine-
tuning of hyperparameters, culminated in a substantial increase in
the overall efficiency of both the LSTM and CNN models, positioning
them as formidable tools in the realm of predictive analytics and
image processing.

8.2 Remaining Task

In the ongoing development of a website aimed at delivering the

best user experience, several critical tasks remain to ensure the
platform’s optimal functionality and user satisfaction. Firstly, the
implementation of responsive design is imperative, ensuring seamless
and visually appealing navigation across various devices, including
desktops, tablets, and mobile phones. This involves meticulous testing
and adjustments to guarantee that the website adapts gracefully to
different screen sizes. Additionally, accessibility features must be
prioritized to make the website inclusive for users with diverse needs.
This involves incorporating alt text for images, providing keyboard
navigation options, and ensuring compatibility with screen readers.

25
REFERENCES

[1] B. Garcia and S. A. Viesca, “Real-time american sign language

recognition with convolutional neural networks,” Convolutional
Neural Networks for Visual Recognition, vol. 2, no. 225-232, p. 8,
2016.

[2] R. H. Abiyev, M. Arslan, and J. B. Idoko, “Sign language trans-

lation using deep convolutional neural networks.,” KSII Transac-
tions on Internet & Information Systems, vol. 14, no. 2, 2020.

26