VISION-BASED GESTURE -

CONTROLLED CURSOR NAVIGATION

SYSTEM
Nehal Ahmed A Mohammed Hashim A
Student, Department of Computer Science Student, Department of Computer Science
and Engineering, and Engineering,
Sathyabama Institute of Science and Sathyabama Institute of Science and
Technology, Technology,
Chennai, India Chennai, India
[email protected] D Sudha [email protected]
Assistant Professor, Department of
Computer Science and Engineering,
Sathyabama Institute of Science and
Technology,
Chennai, India
[email protected]

Abstract—Researchers all throughout the world are Distinguishing between gesture and posture, gesture focuses
working to improve the responsiveness of our gadgets on hand movement, whilst posture highlights the shape of
and to make it possible for them to operate with the the hand. The mouse is a key instrument for Human-
least amount of physical interaction. There are three Computer Interaction (HCI), however in some real-world
primary varieties of mice—wired, wireless, and situations, such Human-Robot Interaction (HRI), it might not
Bluetooth—that need electricity to function. Mice are be practical. Numerous studies have looked at alternative
an essential tool in human-computer interaction. This HCI methods, and hand gestures have come to light as a
study presents a novel computer system that can be natural and intuitive replacement for the conventional
used interactively without a real mouse. The device mouse, especially useful in situations such as human-
recognizes hand motions without any additional machine interfaces. While many people take technology for
hardware thanks to advanced machine learning and granted, utilizing standard gadgets can be challenging for
computer vision algorithms. This approach, which is those who have physical disabilities. Extreme impairments in
compatible with CNN models provided by mediapipe, mobility can severely impair fine motor function, making it
has tremendous potential for a variety of users, difficult to interact with a regular mouse. Sufficient assistive
especially those who struggle with conventional mouse technology are essential for ensuring accessibility for those
operation due to partial paralysis. The technology uses who require it. The idea is to design a virtual mouse system
computer vision to let people use the mouse by showing that replaces touch displays with a camera enabling easier-
various finger counts. Various hand motions are to-use connection with the device. Using vision-based
required for actions like clicking and dragging objects. Computer Vision (CC) can help a webcam reach its full
Additionally, this system uses a camera as an input potential. Using specified labels, this method effectively
device and runs on a single computer. The proposed records and predicts hand motions. The ultimate objective is
solution, which shows the camera's output on the to create a user-friendly virtual mouse system that
screen for user adjustment, utilizes Python and communicates with devices via webcam-based vision,
OpenCV. providing a good replacement for touch displays and
improving accessibility, particularly for those with physical
Keywords- Mediapipe, Convolutional Neural disabilities.
Network, Computer Vision, Hand Recognition, Partial
Paralysis, Human-Computer Interaction, Human- II. LITERATURE REVIEW
Robot Interaction. Researchers and automobile organizations have spent the
previous five years carefully examining the proposed
I. INTRODUCTION problem in an effort to come up with a solution. The
suggested remedies take multiple tacks, focusing on the
Within the field of computer vision, gesture recognition, examination of various patterns.
which focuses on recognizing hand movements as a kind of In order to enhance user-computer interaction, Aashni Hariaa
visual communication has long been an intriguing et al. [1] presented "Hand Gesture Recognition for Human
challenge. These hand gestures involve motions across the Computer Interaction," which makes use of a backdrop
middle of the palm, finger placement, and hand form. Hand extraction and contours recognition algorithm. A variety of
gestures can be categorized as either static or dynamic. sensor modalities, such as gestures, voice, facial expressions,
Static gestures consist of a single, steady hand form, and body language, can be used to facilitate human contact.
whereas dynamic gestures include a series of hand motions, Horatiu-Stefan et al. presented a paper titled "Human Hand
such as waving. Gesture Based System for Mouse Cursor Control." This
model makes use of a computer, a webcam, and a blue hand traditional techniques such as manually entering buttons or
pad. Hand pad postures are used to guide operations. The adjusting the placements of a physical computer mouse.
OpenCV library and Visual C++ 2008 were utilized in the Instead of using a real computer mouse to do similar tasks, it
development of the model. uses computer vision and a camera to monitor and control
A "Mouse Control Using a Web Camera Based on Color different mouse movements.
Detection" was proposed by Abhik Banerjee et al. [2] to Firstly, the hand and its key characteristics are identified by
control click events and cursor movements by sensing the means of Mediapipe. Mediapipe detects a total of 21
color of the image. distinct colors correspond to distinct important points. To recognize the hand, the process uses
cursor controls, and clicking operations are carried out by Mediapipe's palm detection model and hand landmark
simultaneously recognizing these colors. This methodology model. When the hand landmarks are taken into
was developed with the help of MATLAB software and the consideration, palm detection is a simple process.
MATLAB Image Processing Toolbox.
Abhilash et al. [3] introduced the "Virtual Mouse Using
Hand Gesture" approach for manipulating the mouse
pointer and clicking actions. In this example, a red object
was used to build a mask. This approach was implemented
using Python software, which included a number of
modules and functions.
To deal with the mouse pointer and click actions, Alisha
Pradhana et al. [4] proposed the "Design of Intangible
Interface for Mouseless Computer Handling Using Hand
Gestures". The Microsoft Visual Studio integrated
development environment was used to apply this strategy,
which uses a convex hull algorithm. To minimize noise and
identify the user's hand, a red glove is utilized.
"A Real Time Hand Gesture Recognition System Using
Motion History Image" was proposed by Chiung Hsieh et
al. [5] to manipulate the mouse cursor. To minimize the
number of incorrect classifications, the recommended
method makes use of an adaptive skin color detection
model. A C++ software platform with the OpenCV image
processing package loaded was used to construct these
techniques.
"Cursor Control Using Hand Gestures" was the idea put out
by Pooja Kumari et al. [6] to operate a mouse using hand
motions that are recorded by a camera. Using this method,
the camera functions as a sensor to record and detect color
tips applied on the hand. Known as the marker-based
technique, it requires users to hold color-tip mouse cursors Figure 1 Co-ordinates or landmark in hand
in their hands. The MATLAB environment, the MATLAB
1. CAMERA
Image Processing Toolbox, and the OpenCV library were
all utilized in the implementation of this technique. The webcam that is connected to the computer or the
laptop's built-in camera provides the frames that the
system uses to operate. In order to accomplish this, a
III. MATERIALS AND METHODS
video capture object is set up, allowing for real-time
The program relies on specific software requirements
webcam video capturing. Frames are gradually
crucial for its proper operation. These essential
captured by the camera and then sent to the system
prerequisites ensure the functionality, performance, and
for additional processing.
usability of the project:
2. CAPTURING
 Python (version 3.6 – 3.8.5)
Through the use of an infinite loop, the camera
 pycaw captures every frame until the application ends.
 PyAutoGUI Following their extraction from the real-time video,
 AutoPy the frames undergo processing that changes their
 MediaPipe color space from BGR to HSV.
 NumPy
3. COLOR DETECTION AND MASKING
The project development process is based on these Color detection works by identifying color pixels in
predefined software requirements, which ensures the final the webcam frames that correspond to fingertips that
product satisfies anticipated functional, performance, and have colored caps on them. This is the first and
usability criteria. It is essential to remember that, depending crucial stage in the suggested system. This process
on the project's particular objectives and use cases, more yields a grayscale image in which the color cap is
extensive and specialized requirements can be necessitated. highlighted by pixel intensity, setting it out from the
Using a real-time camera to control cursor movement, this surrounding frame. The color cap itself and the
project proposes an innovative approach to Human- rectangle enclosing it are then monitored. Then, by
Computer Interaction (HCI) that offers an alternative to keeping an eye on these color caps, gesture detection
 becomes simpler.

4. GESTURE RECOGNITION
4.1 Mouse Movements
First, by extracting the coordinates of the centers of
the identified rectangles, the computation may be
performed to find the center of the two color objects
that were successfully detected. The built-in
OpenCV function is used to draw a line between
these two locations. Next, the following equation is
used to find the line's midpoint:

M=(Xa+Xb/2, Ya+Yb/2)

The mouse cursor tracks to the identified midway, it

then aligns itself with this midway. In this
configuration, the coordinates that are obtained from
the resolution of the camera-captured frames are Figure 2 Snapshot of Neutral Hand Gesture
transformed to match the resolution of the screen.
The mouse is assigned a preset location, and the
moment the mouse cursor reaches this spot, the
mouse starts functioning. This initiation point is
referred to as the "open gesture," enabling users to
control the mouse pointer effectively.

4.2 Mouse Clicking

In order to execute clicking actions, the suggested
method uses near movements. A new bounding box
forms at the boundary of the tracking bounding
boxes when the bounding boxes of rectangles get
closer to each other. When the newly formed
bounding box has grown to twenty percent of its
original size, the system clicks the left button.
Clicking on the bounding box executes this action.
A double click is made if the user holds this posture
for more than five seconds. The open motion is once
more used to indicate a click of the right button. One Figure 3 Snapshot of Neutral Click Gesture
finger is sufficient to click the appropriate button.
The fingertip is identified by the system by a color
cap, which prompts a right-button click.

4.3 Mouse Scrolling

With this approach, the user uses the open gesture
movement with three fingers, each colored cap, to
begin scrolling. The system performs a scrolling-
down motion when the user simultaneously lines up
three fingers and moves their location downward.

IV. RESULT AND DISCUSSION

In order to contribute to the evolving field of vision-based

machine-human interactions, we have utilized the
principles of computer vision and human-computer
interface (HCI) in our work. Our proposed work is mainly
concerned with hand gesture control of mouse
Figure 4 Snapshot of Mouse Cursor Movement
functionalities. The main features include the ability to
move the mouse, click with the left or right button, double
click, and scroll up or down.
 [6] Pooja Kumari ,Ghaziabad Saurabh Singh, Ghaziabad
Vinay and Kr. Pasi “Cursor Control using Hand Gestures”
International Journal of Computer Applications (0975 –
8887),2016.

[7] Haria, A., Subramanian, A., Asokkumar, N.,

Poddar, S., Nayak, J.S.: Hand gesture recognition for
human computer interaction. Procedia computer
science 115, 367– 374 (2017)
[8] Islam, M.Z., Hossain, M.S., ul Islam, R.,
Andersson, K.: Static hand gesture recognition using
convolutional neural network with data augmentation.
Figure 5 Snapshot of Click Gesture
In: 2019 Joint 8th International Conference on
Informatics, Electronics & Vision (ICIEV) and 2019
V. CONCLUSION
Using a camera, the Virtual Gesture Control Mouse is a
3rd International Conference on Imaging, Vision &
system that can move the mouse pointer and carry out Pattern Recognition (icIVPR). pp. 324–329. IEEE
operations in real time. Mouse navigation, icon selection, (2019)
and related operations like left, right, double click, and
scrolling have all been implemented. To track mouse cursor
[9] Jagannathan, M., Surya, M., BT, A.M.,
movements and icon selection, this system uses motion Poovaraghavan, R.: Finger recognition and gesture
detection and image comparison technologies. Our research based augmented keyboard (2018)
suggests that the system may be made more versatile by
[10] Keil, A., Albuquerque, G., Berger, K., Magnor,
adjusting the algorithms to other areas using a high-quality
camera and enough light. In the future, we want to improve M.A.: Real-time gaze tracking with a consumer-grade
the system by adding additional capabilities including video camera (2010)
interacting with multiple windows, reducing and resizing
windows, and shutting windows. The palm and many
fingers would be used in motion to carry out these
functions. This expansion is envisioned to provide a more
comprehensive and systematized user experience.

REFERENCES
[1] Aashni Hariaa , Archanasri Subramaniana ,
Nivedhitha Asokkumara , Shristi Poddara and Jyothi S
Nayak “Hand Gesture Recognition for Human Computer
Interaction” Proc.ICACC(International Conference on
Advances in Computing & Communications), 2017
August,pp367–374.
[2] Abhik Banerjee, Abhirup Ghosh, Koustuvmoni
Bharadwaj,” Mouse Control using a Web Camera based
on Color Detection”,IJCTT,vol.9, Mar 2019.
[3] Abhilash S , Lisho Thomas, Naveen Wilson, and
Chaithanya “VIRTUAL MOUSE USING HAND
GESTURE” Proc. International Research Journal of
Engineering and Technology (IRJET), e-ISSN: 2395-0056
p-ISSN: 2395-0072,Apr-2018, Volume: 05 Issue:04.
[4] Alisha Pradhana , B.B.V.L. Deepak “Design of
Intangible Interface for Mouseless Computer Handling
using Hand Gestures” ICCCV(International Conference
on Communication, Computing and Virtualization), 2016,
oi: 10.1016/j.procs.2016.03.037.
[5] Chiung Hsieh, D.-Hua Liou, & D. Lee, , “A real time
hand gesture recognition system using motion history
image,” Proc. IEEE Int'l Conf. Signal Processing Systems
(ICSPS), 2.10.1109/ICSPS.2010.5555462, 2020.