INDEX

1. Introduction 2
1.1. Project Details 2
1.2. Project Purpose 2
1.3. Project Scope 2
1.4. Project Goals 3
2. Details of Tools Used 3
2.1. Introduction to OpenCV 3
2.1.1. Eigen Face 5
2.1.2. Fisher Face 6
2.1.3. HAAR Cascade Classifier 8
2.1.4. LBPH Cascade Classifier 9
3. System Analysis 12
3.1. Existing System 12
3.2. Requirements 12
3.3. Installation in Linux 12
4. System Design 13
4.1. Data Flow Diagram 13
5. Screen Shots 14
6. System Implementation 19
6.1. Face Detection and Recognition using LBPH 20
6.2. Real time Face Recognition using Webcam 23
7. System Testing 26
8. System Evaluation 26
9. Limitations and Future Enhancement 26
10. Conclusion and Bibliography 28
11. Verification Procedure 29

1|Page
1. Introduction
The requirement for reliable personal identification in computerized
access control has resulted in an in- creased interest in biometrics.
Biometrics being investigated include fingerprints, speech, signature
dynamics and face recognition. The face is the identity of a person. The
methods to exploit this physical feature have seen a great change since
the advent of image pro-cessing techniques.

The face of the human is a very complex structure and very useful in many
of social aspect in purpose of security. It needs a hard work to identify a
face correctly. Now a day Face Recognition becomes very popular
research area for the researcher. Human have a very powerful organ brain
that can recognize so many faces in a whole life. The researchers are
trying to make a system that can recognize the face in all the
circumstances. A human faces can modify because of many conditions like
increasing age, wearing a glass, have a beard, and change in hair style that
generate complexity in recognition. Although many methods have been
proposed to detect and recognize human face developing a
computational model for a large data base is still a challenging task.

Facial recognition process generally involves two stages:

• Face Detection where the picture is searched to find a face, then the
picture is processed to crop and extract the person’s face for easier
recognition.
• Face Recognition where that detected and processed face is feed into
classifier to decide who that person is.

1.1. Project Details:

This project is about Face Detection and Recognition using dataset of
images and with the help of available classifiers available in OpenCV
python. It involved building a system for face detection and face
recognition using several classifiers available in the open computer
vision library(OpenCV). Face recognition is a non-invasive
identification system and faster than other systems since multiple
faces can be analysed at the same time. The difference between face
detection and identification is, face detection is to identify a face from

2|Page
 an image and locate the face. Face recognition is making the decision
”whose face is it ? ”, using an image database. In this project both are
accomplished using different techniques and are described below. We
will use HAAR-cascades, Eigen face, Fisher face and Local binary
pattern histogram (LBPH) algorithms. Also, we will incorporate the
real time facial detection and recognition part using Webcam.

1.2. Project Purpose:

The main purpose of this mini project is to understand how the face
detection and recognition works in today’s world. Face Recognition
plays an important role in security in modern world. In security, Face
Recognition used in many places i.e. Face unlock in Mobile, PC, ATMs,
Surveillance system etc.

1.3. Project Scope:

The system should be built to at least recognise the people trained
with the dataset images. Also it should detect the person with some
threshold level of confidence. It also needs to detect the face of
person regardless of their face expression, illumination, background,
pose, orientation and occlusion

1.4. Project Goals:

The system should be built to at least recognise the people trained
with the dataset images. Also it should detect the person with some
threshold level of confidence. It also needs to detect the face of
person regardless of their face expression, illumination, background,
pose, orientation and occlusion

2. Details of Tools Used:

2.1. Introduction to OpenCV:
OpenCV is the most popular library for computer vision. Originally
written in C/C++, it now provides bindings for Python.
OpenCV (Open Source Computer Vision) is a library of programming
functions for real-time computer vision. The face detection part of the

3|Page
project was made using an OpenCV Library for Scala. The reason was
that most Face APIs are restricted to doing detection on pictures only,
whereas the project was required to have face detection done on a
live video footage to speed up the process of checking student
attendance and prevent queues before lectures.

The library has more than 2500 optimized algorithms, which includes
a comprehensive set of both classic and state-of-the-art computer
vision and machine learning algorithms. These algorithms can be used
to detect and recognize faces, identify objects, classify human actions
in videos, track camera movements, track moving objects, extract 3D
models of objects, produce 3D point clouds from stereo cameras,
stitch images together to produce a high resolution image of an entire
scene, find similar images from an image database, remove red eyes
from images taken using flash, follow eye movements, recognize
scenery and establish markers to overlay it with augmented reality,
etc. OpenCV has more than 47 thousand people of user community
and estimated number of downloads exceeding 18 million. The library
is used extensively in companies, research groups and by
governmental bodies.

OpenCV uses machine learning algorithms to search for faces within a

picture. Because faces are so complicated, there isn’t one simple test
that will tell you if it found a face or not. Instead, there are thousands
of small patterns and features that must be matched. The algorithms
break the task of identifying the face into thousands of smaller, bite-
sized tasks, each of which is easy to solve. These tasks are also called
classifiers.

OpenCV provides two pre-trained and ready to use face detection

classifiers:
1. HAAR Classifier
2. LBP Classifier
Classifier: A computer program that decides whether an image is a
positive image (face image) or negative image (non-face image) is
called a classifier. A classifier is trained on hundreds of thousands of
face and non-face images to learn how to classify a new image

4|Page
 correctly. Both of these classifiers process images in gray scales,
basically because we don't need color information to decide if a
picture has a face or not.

2.1.1. Eigenfaces face recognizer

This algorithm considers the fact that not all parts of a face are equally
important or useful for face recognition. Indeed, when you look at
someone, you recognize that person by his distinct features, like the
eyes, nose, cheeks or forehead; and how they vary respect to each
other.
In that sense, you are focusing on the areas of maximum change. For
example, from the eyes to the nose there is a significant change, and
same applies from the nose to the mouth. When you look at multiple
faces, you compare them by looking at these areas, because by
catching the maximum variation among faces, they help you
differentiate one face from the other.
It looks at all the training images of all the people as a whole and tries
to extract the components which are relevant and useful and discards
the rest. These important features are called principal components.
Below is an image showing the variance extracted from a list of faces.

Source: https://bit.ly/2I8PJ23

5|Page
 EigenFaces recognizer trains itself by extracting principal components,
but it also keeps a record of which ones belong to which person. Thus,
whenever you introduce a new image to the algorithm, it repeats the
same process as follows:
1. Extract the principal components from the new picture.
2. Compare those features with the list of elements stored during
training.
3. Find the ones with the best match.
4. Return the ‘person’ label associated with that best match
component.
EigenFaces algorithm also considers illumination as an important
feature. In consequence, lights and shadows are picked up by
EigenFaces, which classifies them as representing a ‘face.'

2.1.2. Fisherfaces face recognizer:

This algorithm is an improved version of the last one. Eigen Faces looks
at all the training faces of all the people at once and finds principal
components from all of them combined. By doing that, it doesn't focus
on the features that discriminate one individual from another.
Instead, it concentrates on the ones that represent all the faces of all
the people in the training data, as a whole.

Consider the lighting changes in following images:

Source: https://bit.ly/2I8PJ23

6|Page
Since EigenFaces also finds illumination as a useful component, it will
find this variation very relevant for face recognition and may discard
the features of the other people's faces, considering them less useful.
In the end, the variance that EigenFaces has extracted represents just
one individual's facial features.
Fisher Face Recogniser resolves this issue by extracting principle
components that differentiate one person from others. In that sense,
an individual's components do not dominate (become more useful)
over the others.

Below is an image of principal components using FisherFaces

algorithm.

Source: https://bit.ly/2I8PJ23

FisherFaces only prevents features of one person from becoming

dominant, but it still considers illumination changes as a useful
feature. We know that light variation is not a useful feature to extract
as it is not part of the actual face.

7|Page
2.1.3. HAAR Cascade Classifier:
The Haar Classifier is a machine learning based approach, an algorithm
created by Paul Viola and Michael Jones; which (as mentioned before)
are trained from many many positive images (with faces) and
negatives images (without faces).
It starts by extracting HAAR features from each image as shown by the
windows below:

Source: https://bit.ly/2UxY869
Each window is placed on the picture to calculate a single feature. This
feature is a single value obtained by subtracting the sum of pixels
under the white part of the window from the sum of the pixels under
the black part of the window.Now, all possible sizes of each window
are placed on all possible locations of each image to calculate plenty
of features.

Source: https://bit.ly/2UxY869
8|Page
 For example, in above image, we are extracting two features. The first
one focuses on the property that the region of the eyes is often darker
than the area of the nose and cheeks. The second feature relies on the
property that the eyes are darker than the bridge of the nose.
AdaBoost is a training process for face detection, which selects only
those features known to improve the classification (face/non-face)
accuracy of our classifier.
In the end, the algorithm considers the fact that generally: most of the
region in an image is a non-face region. Considering this, it’s a better
idea to have a simple method to check if a window is a non-face
region, and if it's not, discard it right away and don’t process it again.
So we can focus mostly on the area where a face is.

2.1.4. Local binary patterns histograms (LBPH) Face Recognizer:

Eigen faces and Fisher faces are both affected by light and, in real life,
we can't guarantee perfect light conditions. LBPH face recognizer is an
improvement to overcome this drawback.
The idea with LBPH is not to look at the image as a whole, but instead,
try to find its local structure by comparing each pixel to the
neighbouring pixels.
The LBPH Face Recognizer Process: Take a 3×3 window and move it
across one image. At each move (each local part of the picture),
compare the pixel at the centre, with its surrounding pixels. Denote
the neighbours with intensity value less than or equal to the centre
pixel by 1 and the rest by 0.

Source: https://bit.ly/2UxY869

9|Page
After you read these 0/1 values under the 3×3 window in a clockwise
order, you will have a binary pattern like 11100011 that is local to a
particular area of the picture. When you finish doing this on the whole
image, you will have a list of local binary patterns.

Source: https://bit.ly/2UxY869

After you get a list of local binary patterns, you convert each one into
a decimal number using binary to decimal conversion (as shown in
above image) and then you make a histogram of all of those decimal
values. A sample histogram looks like this:

In the end, you will have one histogram for each face in the training
data set. That means that if there were 100 images in the training data
set then LBPH will extract 100 histograms after training and store
them for later recognition. Remember, the algorithm also keeps track
of which histogram belongs to which person.

Source: https://bit.ly/2UxY869
10 | P a g e
 Later during recognition, the process is as follows:
1. Feed a new image to the recognizer for face recognition.
2. The recognizer generates a histogram for that new picture.
3. It then compares that histogram with the histograms it already has.
4. Finally, it finds the best match and returns the person label associated
with that best match.

Below is a group of faces and their respective local binary patterns

images. You can see that the LBP faces are not affected by changes in
light conditions:

Source: https://bit.ly/2UxY869

11 | P a g e
3. System Analysis:
3.1. Existing System Configuration:
OS Windows 10/ Ubuntu

Processor Intel(R) Core i5-7200U CPU

RAM 8.00 GB

Web Camera 2MP

3.2. Requirements:
 Python 3.3+ or Python 2.7
 Linux or Windows

3.3. Installing on Linux:

First, make sure you have python installed in your OS.
$ python3 –version
Python 3.6
Then install Open CV using Following Command:
$ sudo apt-get install python-opencv

12 | P a g e
4. System Design:
4.1. Flow Chart

13 | P a g e
5. Screenshots
5.1. Face Detection and Recognition Using LBPH

Training Model

14 | P a g e
15 | P a g e
 Prediction

5.2. Real Time Face Recognition using Webcam:

GUI

16 | P a g e
Dataset

17 | P a g e
 Training Model

Real time Prediction

18 | P a g e
6. System Implementation:
To create a complete project on Face Recognition, we must work on 3 very
distinct phases:
1. Face Detection and Data Gathering
2. Train the Recognizer
3. Face Recognition
The below block diagram resumes those phases:

Source: https://bit.ly/2KBjYkK

19 | P a g e
6.1. Face Detection and recognition using LBPH
# Face Recognition with OpenCV
import cv2
import os
import numpy as np

subjects = ["", "Ramiz Raja", "Elvis Presley"]

#function to detect face using OpenCV

def detect_face(img):

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

face_cascade = cv2.CascadeClassifier('opencv-
files/lbpcascade_frontalface.xml')

faces = face_cascade.detectMultiScale(gray,
scaleFactor=1.2, minNeighbors=5);

if (len(faces) == 0):
return None, None
(x, y, w, h) = faces[0]

return gray[y:y+w, x:x+h], faces[0]

def prepare_training_data(data_folder_path):

dirs = os.listdir(data_folder_path)

faces = []
labels = []

for dir_name in dirs:

if not dir_name.startswith("s"):
continue;

label = int(dir_name.replace("s", ""))

20 | P a g e
 subject_dir_path = data_folder_path + "/" +
dir_name
subject_images_names =
os.listdir(subject_dir_path)

for image_name in subject_images_names:

if image_name.startswith("."):
continue;

image_path = subject_dir_path + "/" +

image_name

image = cv2.imread(image_path)

cv2.imshow("Training on image...",
cv2.resize(image, (400, 500)))
cv2.waitKey(100)

face, rect = detect_face(image)

if face is not None:

#add face to list of faces
faces.append(face)
#add label for this face
labels.append(label)

cv2.destroyAllWindows()
cv2.waitKey(1)
cv2.destroyAllWindows()

return faces, labels

print("Preparing data...")
faces, labels = prepare_training_data("training-
data")
print("Data prepared")

print("Total faces: ", len(faces))

21 | P a g e
print("Total labels: ", len(labels))

face_recognizer =
cv2.face.LBPHFaceRecognizer_create()

face_recognizer.train(faces, np.array(labels))

def draw_rectangle(img, rect):

(x, y, w, h) = rect
cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255,
0), 2)

def draw_text(img, text, x, y):

cv2.putText(img, text, (x, y),
cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 255, 0), 2)

def predict(test_img):
img = test_img.copy()
face, rect = detect_face(img)

label, confidence =
face_recognizer.predict(face)
label_text = subjects[label]

draw_rectangle(img, rect)
draw_text(img, label_text, rect[0], rect[1]-5)

return img

print("Predicting images...")

test_img1 = cv2.imread("test-data/test1.jpg")
test_img2 = cv2.imread("test-data/test2.jpg")

predicted_img1 = predict(test_img1)
predicted_img2 = predict(test_img2)
print("Prediction complete")

22 | P a g e
 cv2.imshow(subjects[1], cv2.resize(predicted_img1,
(400, 500)))
cv2.imshow(subjects[2], cv2.resize(predicted_img2,
(400, 500)))
cv2.waitKey(0)
cv2.destroyAllWindows()
cv2.waitKey(1)
cv2.destroyAllWindows()

6.2. Live Face Detection and Recognition using Webcam

6.2.1. Dataset.py
import cv2
#import numpy as numpy

faceDetect =
cv2.CascadeClassifier('haarcascade_frontalface_defau
lt.xml')
cam = cv2.VideoCapture(0)

id = input('Enter ID : ')
sampleNum = 0

while True:
ret,img = cam.read()
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
faces = faceDetect.detectMultiScale(gray, 1.3,
5);
for (x,y,w,h) in faces:
sampleNum = sampleNum + 1
cv2.imwrite("dataset/User." + str(id) + "."
+ str(sampleNum) + ".jpg", gray[y:y+h,x:x+w])
cv2.rectangle(img, (x,y), (x+w, y+h),
(0,0,255), 2)
cv2.waitKey(100)
pass
cv2.imshow("Faces",img)
23 | P a g e
 cv2.waitKey(1)
if sampleNum>100:
break
pass
cam.release()
cv2.destroyAllWindows()

6.2.2. Trainer.py
import os
import cv2
import numpy as np
from PIL import Image

recognizer = cv2.face.LBPHFaceRecognizer_create()
path = 'dataset'

def getImageWithID(path):
imagePaths = [os.path.join(path,f) for f in
os.listdir(path)]
faces = []
IDs = []
for imagePath in imagePaths:
faceImg =
Image.open(imagePath).convert('L')
faceNp= np.array(faceImg, 'uint8')
ID = int(os.path.split(imagePath)[-
1].split('.')[1])
print(ID)
faces.append(faceNp)
IDs.append(ID)
cv2.imshow("training", faceNp)
cv2.waitKey(10)
return IDs, faces

Ids, faces = getImageWithID(path)

recognizer.train(faces, np.array(Ids))
print(faces)
print(recognizer.write('trainingData.yml'))
cv2.destroyAllWindows()

24 | P a g e
6.2.3. Detector.py
import cv2
#import numpy as numpy

faceDetect =
cv2.CascadeClassifier('haarcascade_frontalface_defa
ult.xml')
cam = cv2.VideoCapture(0)
rec = cv2.face.LBPHFaceRecognizer_create()
print(rec.read('trainingData.yml'))
id = 0
font = cv2.FONT_HERSHEY_SIMPLEX

while True:
ret,img = cam.read()
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
faces = faceDetect.detectMultiScale(gray, 1.3,
5);
for (x,y,w,h) in faces:
cv2.rectangle(img, (x,y), (x+w, y+h),
(0,0,255), 2)
id, conf = rec.predict(gray[y:y+h, x:x+w])
if id ==1:
cv2.putText(img,"Shailesh",(x,y+h),
font, 4,(255,255,255),2,cv2.LINE_AA)
elif id ==2:
cv2.putText(img,"Akshay",(x,y+h), font,
4,(255,255,255),2,cv2.LINE_AA)
elif id ==4:
cv2.putText(img,"Varshil",(x,y+h),
font, 4,(255,255,255),2,cv2.LINE_AA)
elif id ==5:
cv2.putText(img,"Hrithik",(x,y+h),
font, 4,(255,255,255),2,cv2.LINE_AA)
else:
cv2.putText(img,str(id),(x,y+h), font,
4,(255,255,255),2,cv2.LINE_AA)
pass

25 | P a g e
 cv2.imshow("Faces",img)
if(cv2.waitKey(1) == ord('q')):
break
pass
cam.release()
cv2.destroyAllWindows()

7. System Testing:
Our system will take 100 real time photos of each person to store in
database and then the dataset of different persons will be used to train
our LBPH model which in turn create histogram for each of the images.
For Face Recognition it will take the real time photo of person from
webcam and convert it to grayscale and then matches with the available
histogram. The least difference histogram will be chosen and the name
of that histogram will be allocated to the person.

We have tested the recogniser in different lighting conditions, poses and

with unknown person samples. The recognizer.predict (), will take as a
parameter a captured portion of the face to be analyzed and will return
its probable owner, indicating its id and how much confidence the
recognizer is in relation with this match.

8. System Evaluation:
We are getting around 30 to 70 % of accuracy depending upon the
surrounding environment which is moderate. Also it depends on the
webcam picture quality as well.

9. Limitations and Future Enhancement:

9.1. Limitations:
We have used LBPH classifier for Face Detection and Recognition
which is good for faster face detection but it is not accurate as
compared to some other available classifiers i.e HAAR Classifier.

26 | P a g e
  Below is the Deference table between HAAR and LBPH Classifier:

Algorithm Advantages Disadvantages

1. Computationally
complex and slow
2. Longer Training time
3. Less Accurate in black
1. High detection Accuracy
HAAR faces
2. Low false Positive Rate
4. Limitations in difficult
lighting conditions
5. Less robust to
occlusion
1. Computationally simple
and fast
1. Less Accurate
2. Shorter training time
LBP 2. High False Positive
3. Robust to local
Rate
illumination
4. Robust to occlusion

Although EigenFaces, FisherFaces, and LBPH face recognizers are fine,

there are even better ways to perform face recognition like using
Histogram of Oriented Gradients (HOGs) and Neural Networks.

9.2. Future Enhancement:

We can add some more features like smile detection, eyes detection,
expression detection, gender detection and detection with some
occlusion i.e. specs, hats, hairstyle, age etc.
The current system can be used in a real time running module like
Arduino Board, Raspberry Pi Board etc.

27 | P a g e
10. Conclusion:
Face recognition technologies have been associated generally with very
costly top secure applications. Today the core technologies have evolved
and the cost of equipment is going down dramatically due to the
integration and the increasing processing power. Certain applications of
face recognition technology are now cost effective, reliable and highly
accurate.

A feature extraction is a quite tricky phase in a process of Recognition.

To get better rate of face recognition the correct choice of algorithm
from many for feature extraction is extremely significant and that plays
significant role in face recognition process. Before selecting the feature
extraction techniques, you must have knowledge of it and which one
performs accurately in which criteria. In this comparative analysis, it is
provided which Feature extraction technique performs accurate in
different criteria.

The project deals with the implementation of a Face Recognition System

Using GUI. Various algorithms for detection and recognition of a face in
a given image have been discussed as a part of the Literature. Here in
this project we are developing a real time GUI based Face recognition
system and further it will be useful for developing a real time face based
Biometric attendance system.

28 | P a g e
 Bhaskaracharya Institute for Space Applications and Geo-informatics
Department of Science & Technology
Government of Gujarat
Phone: 079 - 23213081 Fax: 079-23213091

ISO 9001:2008 E-mail: [email protected], website: www.bisag.gujarat.gov.in

ISO 27001:20013
CMMI LEVEL-5

ISO 27001:2005 Report Verification Procedure

Date: 12th July, 2019

Project Name: Face Recognition using Open CV

Student Name & ID: 1. Mr. Shailesh Arya
2. Mr. Hrithik Mesariya
3. Mr. Akshay Modi
4. Mr. Varshil Rajvir
Soft Copy Hard Copy
Report Format:
Project Index:

Sign by Training Coordinator Sign by Project Guide

29 | P a g e