Minor Project Report Controlling Computer With Hand Gestures Using Arduino and Python
Minor Project Report Controlling Computer With Hand Gestures Using Arduino and Python
Acknowledgment
I would like to express my sincere gratitude to ___________ for providing her invaluable
guidance, comments and suggestions throughout the course of the project. I would also like
to thank her for constantly motivating me to work harder.
3
Table of Contents
Cover 1
Acknowledgment 2
Table of Contents…………………………………………………………………………………………………………………………………… 3
Introducton 4
Circuit Diagram 5
Components Required 5
Hardware Components……………………………………………………………………………………………………………… 5
Sofware Components………………………………………………………………………………………………………………. 5
Arduino6
Technical Specifcatons…………………………………………………………………………………………………………….. 6
Pins……………………………………………………………………………………………………………………………………………. 7
Pin Confguraton………………………………………………………………………………………………………………………. 9
Features……………………………………………………………………………………………………………………………………. 9
Working…………………………………………………………………………………………………………………………………… 10
Applicatons…………………………………………………………………………………………………………………………….. 10
Arduino IDE………………………………………………………………………………………………………………………………………….. 11
Python IDLE………………………………………………………………………………………………………………………………………….. 12
PySerial Library…………………………………………………………………………………………………………………………………….. 13
PyAutoGUI Library……………………………………………………………………………………………………………………………….. 13
Programming Arduino………………………………………………………………………………………………………………………….. 14
Programming Python…………………………………………………………………………………………………………………………… 19
Code……………………………………………………………………………………………………………………………………………………. 22
Conclusion…………………………………………………………………………………………………………………………………………… 25
4
Future Scope………………………………………………………………………………………………………………………………………. 26
References…………………………………………….……………………………………………………………………………………………. 27
5
Introduction
Today we are always surrounded by gadgets. We use mobile phones, computers, tablets, smart
watches all day long and the most important part of using these devices is HMI.
Human Machine Interface or HMI is a system comprising of hardware and software that helps in
communication and exchange of information between the user (human operator) and the
machine. We normally use LED Indicators, Switches, Touch Screens and LCD Displays as a part
of HMI devices.
Another way to communicate with machines like Robots or Computers is with the help of Hand
Gestures. Instead of using a keyboard, mouse or joystick, we can use our hand gestures to control
certain functions of a computer like play/pause a video, move left/right in a photo slideshow, and
scroll up/down in a web page and many more.
Controlling computer using gestures is usually done by using a technology known as Leap
Motion1. The first Laptop to use Leap Motion was Envy 17 Leap Motion SE built by HP ® which
used Leap Motion controller.
Leap Motion is a very expensive technology so in this project we tried to implement the
functionality of controlling a computer with Hand Gestures using the powerful combination of
Arduino and Python.
Instead of using a keyboard, mouse or joystick, we can use our hand gestures to control certain
functions of a computer like play/pause a video, move left/right in a photo slide show, scroll
up/down in a web page and many more, hence we made this project to control VLC Media
Player as hand gestures project.
The idea behind the project is quite easy by using two Ultrasonic Sensors (HC-SR04) with
Arduino. We will place the two Sensors on the top of a laptop screen and calculate the distance
between the hand and the sensor. Counting on the information from Arduino that is sent to the
python through the serial port, this information will be then read by python which is running on
the computer in order to perform certain actions.
Circuit Diagram
To control the PC with Hand Gestures, we connect the two Ultrasonic sensors with Arduino.
The ultrasonic sensors work with 5V and hence they are powered by the on board Voltage
regulator of Arduino. The Arduino can be connected to the PC/Laptop for powering the module
and also for Serial communication.
Components required
Hardware Components:
1. Arduino
2. Two Ultrasonic Sensors (HC-SR04)
3. Connecting Cables
Software Components:
1. Arduino IDE
2. Python IDLE
3. PySerial library
4. PyAutogui library
7
Arduino
Arduino is an open source computer hardware and software company, project, and user
community that designs and manufactures single-board microcontrollers and microcontroller kits
for building digital devices and interactive objects that can sense and control objects in the
physical and digital world.
The Arduino board used for this project is the Arduino Uno.
Technical specifications
Microcontroller: ATmega328P
Operating Voltage: 5v
Input Voltage: 7-20v
Digital I/O Pins: 14 (of which 6 provide PWM output)
Analog Input Pins: 6
DC Current per I/O Pin: 20 mA
DC Current for 3.3V Pin: 50 mA
Flash Memory: 32 KB of which 0.5 KB used by bootloader
SRAM: 2 KB
EEPROM: 1 KB
8
Pins
3V3: A 3.3 volt supply generated by the on-board regulator. Maximum current draw is 50
mA.
GND: Ground pins.
IOREF: This pin on the Arduino board provides the voltage reference with which the
microcontroller operates. A properly configured shield can read the IOREF pin voltage and
select the appropriate power source or enable voltage translators on the outputs to work with
the 5V or 3.3V.
Reset: Typically used to add a reset button to shields which block the one on the board.
Serial: pins 0 (RX) and 1 (TX). Used to receive (RX) and transmit (TX) TTL serial data.
These pins are connected to the corresponding pins of the ATmega8U2 USB-to-TTL Serial
chip.
External Interrupts: pins 2 and 3. These pins can be configured to trigger an interrupt
on a low value, a rising or falling edge, or a change in value.
PWM (Pulse Width Modulation) 3, 5, 6, 9, 10, and 11 Can provide 8-bit PWM output
with the analogWrite() function.
SPI (Serial Peripheral Interface): 10 (SS), 11 (MOSI), 12 (MISO), 13 (SCK). These pins
support SPI communication using the SPI library.
TWI (Two Wire Interface): A4 or SDA pin and A5 or SCL pin. Support TWI
communication using the Wire library.
AREF (Analog REFerence: Reference voltage for the analog inputs.
10
The Ultrasonic transmitter transmits an ultrasonic wave, this wave travels in air and when it gets
objected by any material it gets reflected back toward the sensor this reflected wave is observed
by the Ultrasonic receiver module as shown in the picture below
Now, to calculate the distance using the above formulae, we should know the Speed and time.
Since we are using the Ultrasonic wave we know the universal speed of US wave at room
conditions which is 330m/s. The circuitry inbuilt on the module will calculate the time taken for
the US wave to come back and turns on the echo pin high for that same particular amount of
time, this way we can also know the time taken. Now simply calculate the distance using a
microcontroller or microprocessor.
Applications
Used to avoid and detect obstacles with robots like biped robot, obstacle avoider robot,
path finding robot etc.
Used to measure the distance within a wide range of 2cm to 400cm
Can be used to map the objects surrounding the sensor by rotating it
Depth of certain places like wells, pits etc can be measured since the waves can penetrate
through water.
12
Arduino IDE
The Arduino IDE is incredibly minimalistic, yet it provides a near-complete environment for
most Arduino-based projects. The top menu bar has the standard options, including “File”
(new, load save, etc.), “Edit” (font, copy, paste, etc.), “Sketch” (for compiling and
programming), “Tools” (useful options for testing projects), and “Help”. The middle section
of the IDE is a simple text editor that where you can enter the program code. The bottom
section of the IDE is dedicated to an output window that is used to see the status of the
compilation, how much memory has been used, any errors that were found in the program,
and various other useful messages.
Python IDLE
IDLE is Python’s Integrated Development and Learning Environment.
PySerial Library
PySerial is a library which provides support for serial connections ("RS-232") over a variety of
different devices: old-style serial ports, Bluetooth dongles, infra-red ports, and so on. It also
supports remote serial ports via RFC 2217.
PyAutoGUI Library
The purpose of PyAutoGUI is to provide a cross-platform Python module for GUI automation
for human beings
PyAutoGUI can simulate moving the mouse, clicking the mouse, dragging with the mouse,
pressing keys, pressing and holding keys, and pressing keyboard hotkey combinations
15
Programming Arduino
The Arduino should be programmed to read the distance of hand from the US sensor.
By reading the value of distance we can arrive at certain actions to be controlled with gestures.
For this project we used 5 actions.
Action 1: When both the hands are placed up before the sensor at a particular far distance then
the video in VLC player should Play/Pause.
Action 2: When right hand is placed up before the sensor at a particular far distance then the
video should Fast Forward one step.
Action 3: When left hand is placed up before the sensor at a particular far distance then the
video should Rewind one step.
Action 4: When right hand is placed up before the sensor at a particular near distance and then
if moved towards the sensor the video should fast forward and if moved away the video should
Rewind.
Action 5: When left hand is placed up before the sensor at a particular near distance and then if
moved towards the sensor the volume of video should increase and if moved away the volume
should Decrease.
We start with defining the I/O pins as shown below. The two US sensors are connected to
Digital pins 2, 3, 4 and 5 and are powered by +5V pin. The trigger pins are output pin and Echo
pins are input pins.
The Serial communication between Arduino and python takes places at a baud rate of 9600.
We need to calculate the distance between the Sensor and the hand each time before
concluding on any action. So we have to do it many times, which means this code should be
16
used as a function. We have written a function named calculate_distance() which will return us
the distance between the sensor and the hand.
Inside our main loop we check for the value of distance and perform the actions mentioned
above. Before that we use two variables distL and distR which gets updated with current
distance value.
calculate_distance(trigger1,echo1);
distL =dist; //get distance of left sensor
calculate_distance(trigger2,echo2);
distR =dist; //get distance of right sensor
Since we know the distance between both the sensors, we can now compare it with predefined
values and arrive at certain actions. For example if both the hands are placed at a distance of 40
mc then we play/pause the video. Here the word “Play/Pause” will be sent out through serial
port
if ((distL >40 && distR>40) && (distL <50 && distR<50)) //Detect both hands
{Serial.println("Play/Pause"); delay (500);}
17
If the Right hand alone is placed before the module then we fast forward the video by one step
and if it is left hand we rewind by one step. Based on the action, here the word “Rewind” or
“Forward” will be sent out through serial port
if ((distL >40 && distL<50) && (distR ==50)) //Detect Left Hand
{Serial.println("Rewind"); delay (500);}
if ((distR >40 && distR<50) && (distL ==50)) //Detect Right Hand
{Serial.println("Forward"); delay (500);}
For detailed control of volume and track we use a different methodology so as to prevent false
triggers. To control the volume we have to place the left hand approx. At a distance of 15 cm ,
then we can either move it towards the sensor to decrease the volume of move it away from the
sensor to increase the volume. The code for the same is shown below. Based on the action, here
the word “Vup” or “Vdown” will be sent out through serial port
{
Serial.println("Left Locked");
while(distL<=40)
{
calculate_distance(trigger1,echo1);
distL =dist;
if (distL<10) //Hand pushed in
{Serial.println ("Vup"); delay (300);}
if (distL>20) //Hand pulled out
{Serial.println ("Vdown"); delay (300);}
}
}
}
18
We can use the same method for the right side sensor also, to control the track of the video.
That is if we move the right hand towards the sensor it will fast forward the movie and if you
move it away from the sensor it will rewind the movie. Based on the action, here the word
“Rewind” or “Forward” will be sent out through serial port.
The complete program is then copied to the Arduino IDE and uploaded to Arduino.
19
Programming Python:
The python program for this project is very simple. We just have to establish a serial
communication with Arduino through the correct baud rate and then perform some basic
keyboard actions. The first step with python would be to install the pyautogui module.
cd C:\Python27
Step 2: Inside python directory use the command python –m pip install –upgrade pip to
upgrade pip. Pip is a tool in python which helps us to install python modules easily. Once this
module is upgraded (as shown in picture below) proceed to next step.
Step 3: Use the command “python –m pip install pyautogui” to install the pyautogui module.
Once the process is successful we should see a screen something similar to this below.
Now that the pyautogui module and pyserial module is successful installed with the python, we
can proceed with the python program. The complete python code is given at the end but the
explanation for the same is as follows.
Let us import all the three required modules for this project. They are pyautogui, serial python
and time.
Next we establish connection with the Arduino through COM port. In my computer the
Arduino is connected to COM 4. This is found out by going to Device Manager.
Inside the infinite while loop, we repeatedly listen to the COM port and compare the key
words with any pre-defied works and make key board presses accordingly.
while 1:
incoming = str (ArduinoSerial.readline()) #read the serial data and print
it as line
print incoming
if 'Play/Pause' in incoming:
pyautogui.typewrite(['space'], 0.2)
if 'Rewind' in incoming:
pyautogui.hotkey('ctrl', 'left')
if 'Forward' in incoming:
pyautogui.hotkey('ctrl', 'right')
if 'Vup' in incoming:
pyautogui.hotkey('ctrl', 'down')
if 'Vdown' in incoming:
pyautogui.hotkey('ctrl', 'up')
Code:
Arduino Code:
/*
* Program for gesture control VLC Player
* Controlled uisng Python
*/
const int trigger1 = 2; //Trigger pin of 1st Sesnor
const int echo1 = 3; //Echo pin of 1st Sesnor
const int trigger2 = 4; //Trigger pin of 2nd Sesnor
const int echo2 = 5;//Echo pin of 2nd Sesnor
long time_taken;
int dist,distL,distR;
void setup() {
Serial.begin(9600);
pinMode(trigger1, OUTPUT);
pinMode(echo1, INPUT);
pinMode(trigger2, OUTPUT);
pinMode(echo2, INPUT);
}
/*###Function to calculate distance###*/
void calculate_distance(int trigger, int echo)
{
digitalWrite(trigger, LOW);
delayMicroseconds(2);
digitalWrite(trigger, HIGH);
delayMicroseconds(10);
digitalWrite(trigger, LOW);
time_taken = pulseIn(echo, HIGH);
dist= time_taken*0.034/2;
if (dist>50)
dist = 50;
}
void loop() { //infinite loopy
calculate_distance(trigger1,echo1);
distL =dist; //get distance of left sensor
calculate_distance(trigger2,echo2);
distR =dist; //get distance of right sensor
//Uncomment for debudding
/*Serial.print("L=");
Serial.println(distL);
Serial.print("R=");
Serial.println(distR);
*/
//Pause Modes -Hold
if ((distL >40 && distR>40) && (distL <50 && distR<50)) //Detect both hands
{Serial.println("Play/Pause"); delay (500);}
23
calculate_distance(trigger1,echo1);
distL =dist;
calculate_distance(trigger2,echo2);
distR =dist;
//Control Modes
//Lock Left - Control Mode
if (distL>=13 && distL<=17)
{
delay(100); //Hand Hold Time
calculate_distance(trigger1,echo1);
distL =dist;
if (distL>=13 && distL<=17)
{
Serial.println("Left Locked");
while(distL<=40)
{
calculate_distance(trigger1,echo1);
distL =dist;
if (distL<10) //Hand pushed in
{Serial.println ("Vup"); delay (300);}
if (distL>20) //Hand pulled out
{Serial.println ("Vdown"); delay (300);}
}
}
}
//Lock Right - Control Mode
if (distR>=13 && distR<=17)
{
delay(100); //Hand Hold Time
calculate_distance(trigger2,echo2);
distR =dist;
if (distR>=13 && distR<=17)
{
Serial.println("Right Locked");
while(distR<=40)
{
calculate_distance(trigger2,echo2);
distR =dist;
if (distR<10) //Right hand pushed in
{Serial.println ("Rewind"); delay (300);}
if (distR>20) //Right hand pulled out
{Serial.println ("Forward"); delay (300);}
}
}
}
delay(200);
}
24
Python Code:
import serial #Serial imported for Serial communication
import time #Required to use delay functions
import pyautogui
ArduinoSerial = serial.Serial('com18',9600) #Create Serial port object
called arduinoSerialData
time.sleep(2) #wait for 2 seconds for the communication to get established
while 1:
incoming = str (ArduinoSerial.readline()) #read the serial data and
print it as line
print incoming
if 'Play/Pause' in incoming:
pyautogui.typewrite(['space'], 0.2)
if 'Rewind' in incoming:
pyautogui.hotkey('ctrl', 'left')
if 'Forward' in incoming:
pyautogui.hotkey('ctrl', 'right')
if 'Vup' in incoming:
pyautogui.hotkey('ctrl', 'down')
if 'Vdown' in incoming:
pyautogui.hotkey('ctrl', 'up')
incoming = "";
Make the connections as defined above and upload the Arduino code on your Arduino board.
Then use the python script below and launch the program on your laptop/computer.
Now we can play any movie on your computer using the VLC media player and use hand
gestures to control the movie.
25
Conclusion
Using Arduino, Ultrasonic Sensors and Python we were able to control computer with hand
gestures.
The Ultrasonic Sensors are used to detect the distance of the hands from the sensors or in effect
computer screen since the sensors are mounted on top of the monitor. The ultrasonic sensors
have a Transmitter and a Receiver. The transmitter transmits ultrasonic waves and the Receiver
receives the waves after they are reflected back from an encountered obstacle in the path of the
wave. The obstacles for our purpose are hands.
The Arduino Uno board runs the program used to calculate the distance between the sensors and
the hands. The Python program uses these distances to perform the defined operations on the
VLC media player. The defined operations are Forward, Rewind, Fast Forward, Fast Rewind,
Volume UP, and Volume Down.
This is done using the PySerial and PyAutoGUI libraries. These are used to establish serial
communication with Arduino Uno board.
By executing the Python program and playing a video in VLC, we were able to control the
playback using hand gestures which was the aim of our project.
26
Future Scope
For the purpose of our project we wanted to control the playback of a video in VLC player. We
were able to perform the actions of forwarding, rewinding, fast forwarding, fast rewinding, and
controlling the volume. We could also control the Web Browser using hand gestures by adding a
few lines of codes.
Gestures can be enabled for a lot of programs by changing the codes and we could control a lot
of different types of programs simply by using hand gestures.
The entire HMI system could be better integrated with the computer if the sizes of the
components was even smaller.
Currently hand gestures used for controlling Laptop is done by using Leap Motions.
The Leap Motion controller is a small USB peripheral device which is designed to be placed on a
physical desktop, facing upward. It can also be mounted onto a virtual reality headset. Using two
monochromatic IR cameras and three infrared LEDs, the device observes a roughly
hemispherical area, to a distance of about 1 meter. The LEDs generate pattern-less IR light and
the cameras generate almost 200 frames per second of reflected data. This is then sent through a
USB cable to the host computer, where it is analyzed by the Leap Motion software using
"complex math" in a way that has not been disclosed by the company, in some way synthesizing
3D position data by comparing the 2D frames generated by the two cameras. In a 2013 study, the
overall average accuracy of the controller was shown to be 0.7 millimeters.
The smaller observation area and higher resolution of the device differentiates the product from
the Kinect, which is more suitable for whole-body tracking in a space the size of a living
room. In a demonstration to CNET, the controller was shown to perform tasks such as navigating
a website, using pinch-to-zoom gestures on maps, high-precision drawing, and manipulating
complex 3D data visualizations.
Leap Motion initially distributed thousands of units to developers who are interested in creating
applications for the device. The Leap Motion controller was first shipped in July 2013. In
February 2016, Leap Motion released a major beta update to its core software. Dubbed Orion,
the software is designed for hand tracking in virtual reality.
27
References
Following resources were used for the purpose of this project:
1. www.instructables.com
2. www.circuitdigest.com
3. www.electronicshub.org
4. www.technicalustad.com
5. https://pythonhosted.org/pyserial/
6. www.cs.uky.edu
7. https://en.wikipedia.org
8. www.python.org
9. www.arduino.cc
10. www.theverge.com
11. wiki.python.org
12. www.quora.com
13. docs.python.org
14. www.scribd.com
15. www.digikey.in
16. www.stackexchange.com
17. www.stackoverflow.com
18. www.components101.com
19. www.allboutcircuits.com
20. www.learn.sparkfun.com
21. www.youtube.com