Driver monitoring using the in-vehicle camera

Luka Mašanović Mario Vranješ

Institute RT-RK Osijek LLC for Information Technology Faculty of Electrical Engineering, Computer Science and
Cara Hadrijana 10b, Osijek, Croatia Information Technology
[email protected] Kneza Trpimira 2b, Osijek, Croatia
[email protected]

Radomir Džakula Željko Lukač

RT-RK Institute for Computer Based Systems Faculty of Technical Sciences, University of Novi Sad
Narodnog Fronta 23a, Novi Sad, Serbia Trg Dositeja Obradovića 6, Novi Sad, Serbia
[email protected] [email protected]

Abstract— One of the possible ways to reduce the number of frame of the video signal obtained from the camera located on
accidents in traffic is to track the driver’s condition. In this paper, a the instrument panel and check whether the driver has unusually
new driver monitoring algorithm, which operates on a video captured long-closed eyes, whether the driver looks sideways, and
from the camera located in the instrument panel in the vehicle, is whether the driver’s head has dropped for a long time.
presented. The algorithm examines the presence of the driver's head
in each frame and checks whether the driver has unusually long- The paper is organized as follows. Section II provides the
closed eyes, whether the driver looks sideways, and whether the brief overview of related work. In Section III the details about
driver’s head has dropped for a long time. The algorithm is tested the creation of the new driver monitoring algorithm are given.
using ten different videos recorded under different light conditions. Algorithm performance evaluation is provided in Section IV,
The algorithm generally achieves high performance, while for some whereas last section gives final conclusions of the work.
lighting conditions a certain algorithm upgrading would be necessary
to maintain a high level of accuracy.
II. RELATED WORK
Keywords—driver monitoring, fatigue detection, vehicle This section provides the brief overview of existing driver
monitoring solutions. Generally, driver monitoring systems can
I. INTRODUCTION use four different approaches: (1) the approach based on
biological (physiological) indicators; (2) the approach based on
In today's busy world, where people are under constant the behavior of the vehicle; (3) the approach based on the facial
pressure, stress and other exhausting situations, their ability to analysis and; (4) a hybrid approach. In the first approach, driver
drive can be reduced. According to the World Health biological changes (e.g. heart rate, cerebral waves, blood
Organization statistics, the total number of fatalities per year in pressure) are measured using some sensors attached to the
road traffic is around 1.3 million, while 20 to 50 million get driver. Such systems fall under intrusive systems, because they
injured [1]. The National Sleep Foundation reported that 60 % distract the driver while driving. Additionally, such a system
adult drivers in USA drove while they felt drowsy, and one-third could be easily bypassed by simply not using the sensor to avoid
of them fell asleep while driving [2]. There are several factors a potential warning, which is not a desirable behavior. The
that can contribute to drowsiness. Most important are fatigue approach based on vehicle behavior uses the information
(mental or physical), lack of sleep and opiates consumption [3]. collected from the vehicle, such as unnecessary lane departure,
In order to reduce these figures and detect this kind of driver's vehicle speed, steering wheel movement, etc. The approach
condition in right time, constant driver monitoring is required in based on the facial analysis uses computer vision to determine
the vehicle, so that vehicle can avoid a potential accident. To the condition of the driver. Detecting and tracking the driver's
make driving more comfortable and safer, many different face, eyes, mouth, eye blinking, and cap openings can determine
advanced driver-assistance systems are used. One of them is a potential driver fatigue. These systems are unobtrusive, they do
system that monitors the condition of the driver and it is based not require the wearing of the sensors and they don't distract the
on the analysis of video obtained from the camera located on the driver while driving. In hybrid approach, different methods are
instrument panel in the vehicle. The key functionalities of this combine to determine more precisely whether the driver is
system are driver identification, monitoring of driver alertness careful and tired. Since the new driver monitoring algorithm
and driver attention, with audible warnings. proposed in this paper uses an approach based on the facial
In this paper, the focus is on developing of the new algorithm analysis, below is a brief review of existing solutions that use
that should detect whether the driver's head is present in each the same approach.



    33
 In [4] the authors presented a driving assist system that face profile detector), based on the Viola and Jones algorithm
automatically detects driver fatigue. The system uses algorithms [11] were used. Those detectors are stored as XML files [12].
based on artificial intelligence along with the collected visual Firstly the algorithm tries to detect the front side of the driver's
data from a camera. The system identifies and monitors the face face. If it is not detected, the algorithm tries to detect the face
and eyes, and determines drowsiness using the support vector profile (if the driver is looking sideways). Two different
machines (SVM). In addition to monitoring the blinking of the detectors are used because the characteristics of the face front
eyes, the system takes into account other factors such as and profile are not the same.
yawning, a tilt of the head and face orientation.
In [5], the analysis of driver's eyes and mouth condition is
used. After detecting the face, the multi-block local binary
pattern feature is used to quickly locate and detect eyes. Kalman
filter is used to track the mouth and eyes. After adjusting the
filter and calculating an ellipse that has the shape of a human
eye, then it follows an estimate of the condition of the mouth
and eyes. In assessing the state of the mouth, there is a threshold
which determines the openness and closeness of the mouth, and
yawning of the driver. When checking the eye condition, the
ratio between long and short axis of the previously determined
ellipse is calculated and compared to the given threshold. Test
results have shown that this method can quickly and accurately
detect the position and condition of the eye and mouth.
Driver tracking system that has been successfully tested in
real time and in different light conditions is presented in [6]. The
system estimates whether it is the day or night based on the
brightness of the camera's input image. Depending on whether
the brightness is higher or lower than a certain threshold, a daily
or a night mode of the algorithm is used, respectively. For the
daily operation mode, AdaBoost method is used to detect face
and eyes by drawing rectangles around them. After eyes
detection, characteristic parameters such as the height and width
of the eye and the curvature of the upper lid are calculated.
Based on this data, driver fatigue is determined using the
PERCLOS method, which determines the state of the driver Fig. 1. Flow chart of the proposed algorithm for driver monitoring
based on the closure of the lids. Night mode uses the difference
between infrared frames from the camera to detect the eyes. In After the face (the front or the profile) is detected for the first
addition to the eyes, as an additional indicator of fatigue, the time, another function which executes only once, is called. This
driver's yawning is also observed. After detecting the eyes, it is function stores the initial data for the detected face, i.e. the size
possible to locate the mouth position of the face structure. To of the head and the coordinate of the upper left corner for
test the yawning of the driver, the LPB (local binary pattern) checking the head position. Head size is calculated multiplying
algorithm was used. the height and width of the rectangle which marked the face,
expressed in pixels. This initial data will later serve for checking
More driver monitoring solutions based on different whether the driver’s head is down and whether the head distance
approaches (not only those based on the facial analysis can be from the camera is changed. Note that, if there are more people
found in [7-10]. in the frame (e.g. passengers behind the driver), the algorithm
will detect the biggest face (i.e. face closest to the camera).
III. PROPOSED ALGORITHM FOR DRIVER MONITORING The eyes are looked for only if the front side of the face is
In this section the new proposed algorithm for driver detected. For that purpose, the pre-trained eye detector based on
monitoring, based on facial analysis is described. The algorithm the Viola and Jones algorithm is used. The eyes are not searched
examines whether the driver is present in the scene obtained in the entire frame but only in the part where detected face is
from the camera located in the instrument panel, whether the located. In Fig. 2 (a) the successfully detected face and eyes are
driver is sleeping, looking sideways, or keeping the head down. shown.
The algorithm was created using OpenCV library and C++
programming language. The flow chart of the algorithm if given B. Checking eyes condition
in Fig. 1, and more details are given in rest of this section. Every time the face is detected and the eyes are not, a
function which measures eye closure time is called. The counter
A. Face and eyes detection measures how long the eyes are not detected. If the counter
The first step of the algorithm is to examine whether the exceeds a certain threshold, an alarm will be triggered, which
driver’s head is present in the current frame. For that task, two implies that the driver is likely sleeping. In Fig. 2 (b) the case of
different pre-trained detectors (the front face detector and the the face with closed eyes is shown (eyes are not detected).

34
 face profile, the separate detector is required for each side of the
face. To reduce the use of more unnecessary detectors, only the
left face profile detector was used. The right-side face profile
detection is solved by using an engineering approach. Namely,
if the left profile is not detected, the algorithm will call the
function that will rotate the image around the y axis, i.e. the
mirror effect will be achieved. With this approach it is possible
to detect the left and right side of the face using only one
detector. Figure 5 shows face detection for the right and left
profile of the face.

(a) (b)
Fig. 2. (a) Successful face and eye detection (b) successful face detection and
the eyes are closed (not detected) more than a predefined period.

C. Head down check

The next check which is performed is head down check. It
compares the y coordinate of the upper left corner of initial face
rectangle with the y coordinate of the upper left corner of face
rectangle in current frame. If the driver lowers his head down,
the y coordinate decreases. Furthermore, if the driver keeps the
head down longer than the predefined period, the alarm is (a) (b)
activated. Normal and lowered head state is shown in Fig. 3. Fig. 4. The change of the face rectangle size and y coordinate of the upper left
corner of face rectangle (a) the driver is closer to the camera (e.g. initial
position), the larger size of the face rectangle (b) the driver has moved away
from the camera, smaller the size of the face rectangle.

(a) (b)
Fig. 3. The driver’s head in the (a) normal position (b) lowered position

D. Head position check

(a) (b)
The driver can move closer or further from the camera while Fig. 5. Face profile detection when the head is facing to the (a) right (b) left.
driving (by moving the seat or changing the seating position). In
that case the value of the y coordinate of the upper left corner of
face rectangle is also changed and could falsely indicate the IV. EXPERIMENTAL RESULTS
driver’s head dropping. To avoid such a mistake, the head To check the performance of the new driver monitoring
position check is implemented in the new algorithm, which algorithm, ten different videos were recorded in the real
additionally compares the size of the initial face rectangle to the environment (camera located in the instrument panel in the
size of the face rectangle in the current frame. If the size is larger vehicle in front of the driver, both men and women). For this
or smaller than this of initial size (the certain threshold is test the recording was done with the Samsung Galaxy S6 front
defined), the face rectangle size and y coordinate of the upper camera. The videos were originally recorded during the day
left corner of face rectangle are updated to the current size and under different light conditions, with resolution of 1280x720
position (value). In Fig. 4, the case when the driver moves away pixels, then down sampled to 640x480 pixels prior analysis.
from the camera and the data are updated is shown.
In Table I the results of the algorithm performance
evaluation are presented. Specifically, it is shown how many
E. Detection and verification of the face profile
times the algorithm has successfully detected a particular action
If the algorithm cannot detect the front side of the face, it will (driver looking on the side, driver kept his eyes closed, and
try to detect the profile of the face. If the face profile is detected, driver lowered his head relative to the normal head position
the timer starts and counts how long the driver looks to the side. longer than 2 seconds) relative to how many times the action
If the counter exceeds a certain threshold, a warning alert is was really performed. In Figures 6-9 the examples of successful
activated, which indicates that the driver is looking to the side and unsuccessful detections are presented.
too long. Since the used pre-trained detector can only detect one

35
TABLE I. RESULTS OF THE PROPOSED DRIVER MONITORING ALGORITHM
PERFORMANCE EVALUATION USING TEN DIFFERENT VIDEO SIGNALS RECORDED
IN DIFFERENT LIGHT CONDITIONS

Looking on the Eyes closed Head down

side accurately accurately accurately
detected / total detected / detected /
VIDEO number of looks total number total number
on the side of eyes closed of head down
(a) (b)
Video 1 4/6 1/2 3/3
Fig. 7. Examples of successful detections when the driver (a) turns the head to
Video 2 1/2 2/2 1/2 the right (b) turns the head to the left

Video 3 2/2 1/1 0/1

Video 4 4/4 1/2 1/2
Video 5 1/2 1/1 2/2
Video 6 4/4 3/3 2/3
Video 7 3/3 1/1 1/1
Video 8 3/3 2/2 2/2
Video 9 2/2 1/2 2/2
Video 10 4/6 4/5 3/4 (a) (b)
Fig. 8. Examples of successful detections when the driver keeps the eyes closed
Total: 28 / 34 17 / 21 17 / 22 (a) person A (b) person B
Success: 82.35% 80.95% 77.27%

(a) (b)
Fig. 9. Examples of successful detections when the driver’s head is dropped (a)
(a) (b) person B, (b) person D

(c) (d)
Fig. 6. Examples of successful face and eye detections for different persons, (a) (b)
when the driver is looking ahead and the eyes are opened (a) person A, (b)
person B, (c) person C, (d) person (D)

From Table I, when considering the results for all videos

together, it can be seen that for all three action types the action
detection accuracy of the proposed algorithm is 80+3 %. By
analyzing those results, it was concluded that the accuracy of the
algorithm for particular action is mostly influenced by the
minNeighbors parameter of the OpenCV function
detectMultiScale(), that performs particular detection using pre- (c)
trained detectors (face/eye). Fig. 10. Examples of unsuccessful detection: (a) of the driver’s face and eyes (b)
when the driver’s head is dropped (c) false detection of front face and eyes

36
 Namely, minNeighbors parameter specifies how many More details about minNeighbors parameter can be found in
neighbors (rectangles) each candidate face rectangle should [13]. In order to ensure greater precision and avoid
have to retain it as positive, and as such has the greatest misinterpretation for detectors used in our algorithm, values of
influence on detection accuracy. The idea behind this parameter minNeighbors parameter for front side of the face and eye
is that the detector will run on a multiple scale style and at the detection are set to 15 and 20, respectively.
same time following a sliding window strategy. After this step,
it will give multiple responses even for a single face region. This Sleep detection depends on the detection of the eyes, whose
parameter tends to filter these responses just like by setting up a detection depends on whether the face is detected or not. An eye
lower-bound threshold, i.e. it will only be counted as a valid face detection detector is trained to mark only the open eye. By using
if the number of responses for this face is higher than such a detector when the driver’s eyes are closed, the eyes
minNeighbors. Higher value of minNeighbors results in fewer should not be detected. During the development of the algorithm
detections but with higher quality (eliminates false positives), it was discovered that sometimes the eye detector marked the
whereas at the same time some true positives can be undetected driver's eyes even when they are closed. It results in restarting
due to high number of minNeighbors required. On the other the counter which should alert the driver to wake up. To avoid
hand, lower values of minNeighbors results in more false such situations as with face detection, the minNeighbors is set
positive face detections. To illustrate the effect of minNeighbors to higher value (20) for eye detection. Furthermore, regarding
parameter on the accuracy of detection of desired object, i.e. eye detection, it is interesting to see in Fig. 6 (d) that only one
face, some examples are given in Fig. 11. eye of the driver is detected. The second eye is not detected due
to driver’s glasses and reflection of the light on the glass in front
of undetected eye.
On the other hand, for face profile detection, minNeighbors
parameter is set to 3 since it achieves the best accuracy. As soon
as the driver turns his head to the side, the front face detection
should not happen. Furthermore, the front face detector
accuracy was also influenced by the surrounding light. Namely,
if the bright light shone on the driver’s face (such as in Fig. 10
(a) and (b)), due to its mode of operation, the detector would not
successfully mark it. This problem could be resolved by
performing some image preprocessing techniques, prior
application of the algorithm itself.
(a) Regarding results for head down detection, one of the
reasons for mistakes is that as when driver lower his head, the
focus on the face is lost and the detector does not detect the face.
Another reason is the high value of minNeighbors parameter for
front face detector, which is used here. As a solution to this
problem, it is possible to train a new separate detector which
should detect the top of the head or when the head is in the
lowered position.

V. CONCLUSION
This paper presents a new driver monitoring algorithm
operating on the signal obtained from the camera located in the
(b) instrument panel in the vehicle. The algorithm uses face and eye
detectors pre-trained according to Viola and Jones algorithm
and is developed to detect whether the driver's head is present in
each frame of the video and check three cases: whether the
driver has unusually long-closed eyes, whether the driver looks
sideways, and whether the driver’s head has dropped for a long
time. The algorithm is tested on then different video sequences
and the results are on a satisfying level, but could be improved
by performing some additional operations. The possible
improvements could be achieved by training detectors on the
bigger data set, training additional detector when head is in
lowered position, training the detector for eyes when driver is
wearing eyeglasses and adding a case for driver monitoring in
(c)
night. All of them would be included in our future work.
Fig. 11. Face detection using pre-trained detector for face detection and OpenCV
function detectMultiScale() with different value of the function parameter
minNeighbors (a) minNeighbors = 0 (b) minNeighbors = 1 (c) minNeighbors =
3 (optimal for this image)

37
 ACKNOWLEDGEMENT
This work was partially supported by J.J. Strossmayer
University of Osijek business fund through the internal
competition for the research and artistic projects „UNIOS ZUP-
2018“ and partially by the Ministry of Education, Science and
Technological Development of the Republic of Serbia, under
grant number III44009-1.

REFERENCES
[1] World Health Organization, Global status report on road safety, 2009.
[2] The National Sleep Foundation, Sleep in America™ Poll, 2009, Available
at: https://www.sleepfoundation.org/sleep-topics/drowsy-driving-vs-
drunk-driving-how-similar-are-they
[3] W. Vanlaar, H. M. Simpson, D. Mayhew, and R. Robertson, „Fatigued and
drowsy driving: attitudes, concerns and practices of Ontario drivers“,
Traffic Injury Research Foundation, 2007.
[4] M. J. Flores, J. M. Armingol and A. Escalera, „Real-time warning system
for driver drowsiness detection Using Visual Information”, Journal of
Intelligent & Robotic Systems, August, 2010.
[5] X. Tang, P. Zhou, P. Wang, „Real-Time Image-based Driver Fatigue
Detection and Monitoring System for Monitoring Driver Vigilance“, IEEE,
27 – 29 july, 2016.
[6] X. Luo, R. Hu, T. Fan, „The driver fatigue monitoring system based on face
recognition technology“, IEEE, 9 – 11 june, 2013.
[7] I. Park, J. Ahn, H. Byun, „Efficient Measurement of the Eye Blinking by
Using Decision Function for Intelligent Vehicles“ , Computational Science
– ICCS, 2007
[8] A. Koesdwiady, R. Soua, F. Karray, M. S. Kamel, „Recent Trends in Driver
Safety Monitoring Systems: State of the Art and Challenges,” IEEE
Transactions on Vehicular Technology, vol. 66, no. 6, 2017, pp. 4550-4563.
[9] M. Choi, G. Koo, M. Seo, S. W. Kim, „Wearable Device-Based System to
Monitor a Driver’s Stress, Fatigue, and Drowsiness,” IEEE Transactions
on Instrumentation and Measurement, vol. 67, no. 3, 2018, pp. 634-645.
[10] Q. Zhang, Y. Zhou, S. Song, G. Liang, „Haiyang Ni, Heart Rate Extraction
Based on Near-Infrared Camera: Towards Driver State Monitoring,” IEEE
Access, vol. 6, 2018, pp. 33076-33087.
[11] P. Viola, M. Jones, „Robust Real-Time Face Detection”, International
Journal of Computer Vision, vol. 57, no. 2, 2004, pp. 137-154
[12] Github, openCV detectors. Available:
https://github.com/opencv/opencv/tree/master/data/haarcascades
[13] OpenCV, detectMultiScale() function. Available:
https://docs.opencv.org/2.4/modules/objdetect/doc/cascade_classification.
html#cascadeclassifier-detectmultiscale

38