IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 25, NO.

5, MAY 2021 1409

Camera-Based Vital Signs Monitoring During

Sleep – A Proof of Concept Study
Mark van Gastel , Sander Stuijk , Sebastiaan Overeem , Johannes P. van Dijk ,
Merel M. van Gilst, and Gerard de Haan

Abstract—Polysomnography (PSG) is the current gold gold-standard tool for the diagnosis of sleep apnea. A collection
standard for the diagnosis of sleep disorders. However, of sensors is used to capture the physiological information
this multi-parametric sleep monitoring tool also has some required for a reliable diagnosis, e.g. electroencephalography
drawbacks, e.g. it limits the patient’s mobility during the
night and it requires the patient to come to a specialized (EEG), respiration and oxygen saturation. The requirement of
sleep clinic or hospital to attach the sensors. Unobtrusive attaching many wired sensors to the body has many drawbacks,
techniques for the detection of sleep disorders such as e.g. it limits the patient’s mobility during the night and it requires
sleep apnea are therefore gaining increasing interest. Re- the patient to come to a specialized sleep clinic or hospital to
mote photoplethysmography using video is a technique
attach the sensors. Both factors result in that the observational
which enables contactless detection of hemodynamic in-
formation. Promising results in near-infrared have been re- study does not reflect a normal night’s sleep, and hence could
ported for the monitoring of sleep-relevant physiological affect the diagnostic value of PSG. This motivated the research
parameters pulse rate, respiration and blood oxygen sat- on unobtrusive and remote methods for apnea detection, such as
uration. In this study we validate a contactless monitoring watches [3], depth cameras [4], radars [5], thermal cameras [6]
system on eight patients with a high likelihood of relevant and mattresses [7]. Although not a replacement for all sensors of
obstructive sleep apnea, which are enrolled for a sleep
study at a specialized sleep center. The dataset includes a full PSG, the surrogate measurements of the proposed methods
46.5 hours of video recordings, full polysomnography and show promise, e.g. for pre-screening of sleep apnea. However,
metadata. The camera can detect pulse and respiratory rate the methods are often a replacement for only a single PSG sensor,
within 2 beats/breaths per minute accuracy 92% and 91% of whereas for a reliable diagnosis one would like to measure
the time, respectively. Estimated blood oxygen values are multiple physiological parameters which are affected by apnea.
within 4 percentage points of the finger-oximeter 89% of the
time. These results demonstrate the potential of a camera Furthermore, the size and costs of some of these methods could
as a convenient diagnostic tool for sleep apnea, and sleep hamper wide-spread adoption.
disorders in general. Photoplethysmography (PPG) is a rich source of hemody-
Index Terms—Contactless monitoring, sleep apnea,
namic information and has the potential of replacing multiple
sleep diagnostics, near-infrared. PSG sensors, e.g. for the measurement of heart/pulse rate, res-
piration and blood oxygen saturation. Camera-based, or, remote
PPG enables simultaneous contactless monitoring of all these
I. INTRODUCTION parameters from the skin with a single sensor modality, without
LEEP apnea is one of the most common sleep disorders the need of attaching probes to the patient’s body [8]–[10]. Fur-
S affecting an estimated 9–21% of women and 24–31% of
men worldwide [1], [2]. Polysomnography (PSG) is the current
thermore, using infrared illumination, patients can be monitored
in darkness. The application of this technique during sleep is
challenging as the visible skin, required for remote PPG, is
often limited and furthermore unpredictable in appearance, e.g.
Manuscript received May 11, 2020; revised October 3, 2020 and due to occlusion and sleep position. To overcome this problem
November 17, 2020; accepted December 13, 2020. Date of publication
December 18, 2020; date of current version May 11, 2021. This work we developed a system based on remote PPG features and
was supported by the framework of the IMPULS-II Program for the Data validated it on a limited dataset consisting of recordings of
Science Flagship Project. (Corresponding author: Mark van Gastel.) healthy subjects who were instructed to simulate realistic sleep
Mark van Gastel is with the Philips Research, Eindhoven 5656,
Netherlands, and also with the Department of Electrical Engineering, scenarios [11], e.g. different sleep positions. It was shown that
Eindhoven University of Technology, 5612 Eindhoven, The Netherlands the proposed system outperforms the current state-of-the-art
(e-mail: [email protected]). for this use case, with mean absolute errors of 1.85 beats per
Sander Stuijk and Gerard de Haan are with the Department of Electri-
cal Engineering, Eindhoven University of Technology, 5612 Eindhoven, minute and 4.11 percentage points for pulse rate and oxygen
The Netherlands (e-mail: [email protected]; [email protected]). saturation, respectively. The aim of this proof-of-concept study
Sebastiaan Overeem, Johannes P. van Dijk, and Merel M. van Gilst is to evaluate whether the typically contact-based measured pa-
are with the Department of Electrical Engineering, Eindhoven University
of Technology, 5612 Eindhoven, The Netherlands, and also with the rameters pulse rate, respiratory rate and peripheral arterial blood
Kempenhaeghe Center for Sleep Medicine, 5591 Heeze, The Nether- oxygen saturation (SpO2 ) can be accurately measured remotely
lands (e-mail: [email protected]; [email protected]; with a camera in obstructive sleep apnea (OSA) patients who
[email protected]).
Digital Object Identifier 10.1109/JBHI.2020.3045859 undergo an overnight PSG at a specialized sleep center, where

2168-2194 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 1410 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 25, NO. 5, MAY 2021

Fig. 2. Optical characteristics of the camera system: camera sensi-

tivity (black), responses of the 760/800/889 nm filters (blue), and near-
infrared illumination (red).
Fig. 1. Illustration of the camera setup installed in the bedroom. Three
identical cameras with different optical filters capture the scene simulta-
neously with the full PSG. Near-infrared illumination units are placed at TABLE I
both sides of the cameras. A signal generated by the video system is OVERVIEW OF THE PSG SENSORS USED FOR VALIDATION
transmitted to the PSG head box to synchronize the data streams from
the two independent acquisition systems.

we compare the remotely sensed physiological signals with the

PSG sensory data. The contributions of this study compared to
our previous study [11] are; 1) the recordings are from OSA
patients with PSG sensors attached to the skin and body, 2) the
recordings are not short segments but cover the entire night, and
3) the physiological variations in a population with disturbed
breathing are much larger compared to healthy subjects. To the
best of the authors’ knowledge, this is the first time that such a
study is conducted.
The remainder of the paper is structured as follows. In
Section II, the experimental setup installed at the sleep center,
the processing framework and details about the studied popula- Fig. 3. Protocol used to synchronize the PSG and video data.
tion are discussed. The results are presented in Section III. In
Section IV the results are discussed including suggestions for
future work. Finally, conclusions are drawn in Section V. can be observed from Fig. 2. The bulbs are installed in two lamp
holders, type LHD-B628FS of Falcon Eyes, placed at both sides
II. MATERIALS AND METHODS of the cameras. Cameras and illumination are attached to a tube
placed above the patient’s head and connected to the ceiling.
A. Experimental Setup
The cameras are externally triggered via a National Instruments
The setup comprises cameras, data acquisition equipment NI6251 DAQ. An acquisition notebook is used to control the
and near-infrared (NIR) illumination, as visualized in Fig. 1. DAQ via a USB interface and to store the video data via an
The cameras are three identical monochrome cameras, type ethernet interface. The DC power supply for the illumination can
Manta G283B of Allied Vision GmbH, and are equipped be controlled remotely via the same notebook. To keep file sizes
with three different optical filters to obtain spectral selectivity manageable as the data is stored uncompressed, the video data
in NIR. The three bandpass filters from Semrock, Inc. have is stored in segments with a maximum duration of 5 min. The
Center-Wavelenghts (CWLs) of 760 nm, 800 nm and 889 nm, PSG data is captured by the Grael 4K from Compumedics, from
with Guaranteed-Minimum-Bandwidths (GMBWs) of 12 nm, which a subset of the data is used for analysis, as listed in Table I.
12 nm and 42 nm, respectively. All cameras are equipped with 1) Synchronization: The PSG data and video data are cap-
12.5–75 mm manual zoom-lenses, type M6Z1212-3S of Com- tured by two independent acquisition systems, as visualized in
putar, where the zoom-factor is selected such that the cameras Fig. 1. In order to synchronize the video data with the PSG data,
capture the full width of the bed to not make assumptions on a video synchronization signal is transmitted from the video
the sleep position. The cameras capture at a stable frame rate acquisition system to the PSG device. The protocol of the signal
of 15 fps, with a resolution of 728 × 968 and with 8-bits depth. is visualized in Fig. 3. For each time-interval tsync (default: 10 s),
Frame alignment of the three cameras is performed off-line using a 32-bits word with the number of captured frames is encoded
an affine transformation. Illumination is provided with incandes- in an analogue signal and transmitted to an input channel of
cent light bulbs with a visible light blocking coating (Exo Terra the PSG device via an XLink interface, where it is sampled at
Night Heat Lamp), which provide a broad emission spectrum 64 Hz in parallel with the other PSG sensors. After the video
in NIR covering the transmittances of all three optical filters, as registration, the synchronization signal is decoded and used

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 VAN GASTEL et al.: CAMERA-BASED VITAL SIGNS MONITORING DURING SLEEP – A PROOF OF CONCEPT STUDY 1411

Fig. 4. Processing framework for the vital signs extraction from the video data.

to link the video segments with the PSG data. The remaining
synchronization error is smaller than the time interval between
two frames, 66 ms with our frame rate of 15 fps.

B. Processing
To extract the physiological information from the video data
we use the processing pipeline previously developed for sleep Fig. 5. Available video data per sleep stage (left) and sleep position
applications, validated on healthy volunteers [11]. The pipeline (right) in the dataset.
can be divided into five steps as visualized in Fig. 4: 1) pre-
processing, 2) parallel pulse-extraction, 3) similarity mapping,
4) region-of-interest selection, and 5) vital signs extraction. For
more details on the framework we refer to [11]. The passbands
of the 6th -order Butterworth bandpass filters are in the range
35–240 beats per minute (BPM) for pulse and oxygen saturation,
and 8–35 breaths per minute (BRPM) for respiration. For the
SpO2 estimation we used the static pulse vector [0.39 0.54 0.73]
and the update vector [1 −0.05 −0.36] [12], similar to our earlier
experiments [11]. Fig. 6. Size of the Region-of-Interest (ROI) as fraction of the total
frame size.
C. Population
In this study eight subjects (five males) with sleep complaints technologists according to the American Academy of Sleep
were enrolled. These subjects were scheduled to undergo a noc- Medicine (AASM) scoring rules [14]. The dataset contains valid
turnal observational study with full PSG at sleep center Kempen- video recordings with a total duration of 46.5 hours. Reasons for
haeghe (Heeze, The Netherlands) and have been selected based invalid recordings include the absence of illumination caused by
on a suspicion of relevant OSA. Exclusion criteria include other the operator, and corrupted files. The duration of the video data
expected sleep pathologies and severe co-morbidities (e.g. neu- per sleep stage and position is displayed in Fig. 5.
rological, cardiovascular and psychiatric disorders). The sub-
jects were in the age range 45 − 69 years (average: 58.6 ± 8.9), III. RESULTS
had BMIs in the range 25.5 − 31.6 (average: 28.0 ± 1.88), had This section presents the results obtained on the dataset. Each
Apnea Hypopnea Indices (AHIs) in the range 7.7 − 52.7 (aver- monitored physiological parameter is discussed separately. The
age: 24.1 ± 18.1) and were all Caucasian. The study has been sizes of the region-of-interests (ROIs) of all patients for the
approved by the Philips Institutional Review Board (Internal different sleep positions are displayed in Fig. 6, where the sleep
Committee on Biomedical Experiments) and the Kempenhaeghe positions ‘upright’ and ‘front’ were omitted as these are not
Review Board before experimentation. Informed consent was present in the dataset. In this section we present the overall
obtained from each patient prior to the measurement. Infor- results of all patients. The oxygen saturation results of each
mation about the sleep registrations are visualized in Fig. 20 patient are presented in Appendix I.
and Table VII of Appendix I, showing Rapid Eye Movement
(REM) sleep as well as the 3 deepening stages of non-REM
sleep (N1 to N3). Each of these stages has specific physiological A. Pulse Rate
and movement characteristics [13], e.g. during REM sleep the The results for the cardiac pulse signal grouped by sleep stage
skeletal muscles are paralyzed with the exception of occasional and sleep position are summarized in Table II. We computed the
muscle twitches. An operator in an adjacent monitoring room evaluation metrics with analysis windows of 5 s, 10 s and 20 s,
initiated the recordings and controlled the illumination to not and used the signal from the ECG electrodes for validation. To
interfere with the sleep registration. Recordings were interrupted compare the performance of the camera with its PSG counter-
when the operator suspected that the video registration affected part, we also calculated the evaluation metrics for the finger-PPG
sleep and/or the patient left the bed. Besides the full PSG sensor. The pulse rates are determined by transforming the pulse
registrations, body/head positions and hypnograms are available signals to the frequency domain and selecting the peak-energy
for each 30 s epoch. PSGs were scored by experienced sleep locations. The heart-rates from the ECG signal are determined

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 1412 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 25, NO. 5, MAY 2021

TABLE II
RESULTS OF THE CAMERA-DERIVED PULSE SIGNAL FOR DIFFERENT LENGTHS OF THE ANALYSIS WINDOW

Fig. 7. The percentage of time (PERC) where the pulse measurement

error is within the range 1–10 BPM, grouped per sleep stage and posi-
tion. The first two figures show the results for the camera pulse signal,
the third and last that of the finger-PPG signal. All use the ECG-derived
heart rate as reference.
Fig. 9. Examples of the pulse signals for the scenarios: (left) a change
in sleep position, (middle) apneic events, and (right) movements. All ex-
amples are segments of 5 minutes and computed with analysis windows
of 20 s.

temporarily is not capable to extract the pulse signal, but is able to

quickly recover. Also the finger-PPG measurement is corrupted
during the change in position. The second example shows the
performance during an apneic event where the heart is stressed,
which increases blood pressure and heart rate. The third example
is a segment with body movements. It can be observed that the
camera pulse signal is much noisier, although cleaner than the
finger-PPG sensor.
Fig. 8. The duration distributions of unreliable segments for different
accuracy criteria, where the integer values indicates the pulse rate error
in beats per minute (BPM). The inset is a zoomed version of the lower B. Respiration
part of the graph.
The respiration results are summarized in Table III. We
compared the camera-based respiration signal with a reference
by using a peak-detector in the time-domain for the localisation breathing rate derived from the respiratory PSG data, MedResp:
of the R-peaks. In line with the work of De Haan and Jeanne r PSG RIP = PSG Thorax + PSG Abdomen
[15], the remote PPG signal is quantitatively measured by the r MedResp = median(BRPSG RIP ,BRPSG Flow ,BRPSG Thermistor ).
signal-to-noise ratio (SNR). Besides SNR, we calculated the First, the signals from the two respiration bands attached to
mean-absolute-error (MAE), root-mean-squared-error (RMSE) the thorax and abdomen are combined, which reflect respiratory
and bias (B) for evaluation. The percentage of time (PERC) effort. The sum is taken to be independent of breathing type
where the measurement is accurate within the range 1–10 BPM and/or to be more robust to sensor displacement during the night.
for an analysis window of 10 s is visualized in Fig. 7. Fig. 8 The reference used for validation (MedResp) is the median of
displays the duration distributions of unreliable segments for the breathing rates (BR) from the respiration bands and the other
different accuracy criteria, where the integer values indicate two PSG respiratory sensors, flow and thermistor. We calculated
the error in BPM. Examples of the extracted pulse signal for the median to create a reliable reference, which is less affected
different scenarios are visualized in Fig. 9. The first example by artifacts and/or sensor displacements. Similar to pulse, we
includes a change in sleep position from the back to the left calculated the metrics MAE, RMSE and B for the different
side. From the spectrograms it can be observed that the pulse sleep stages and sleep positions. The results are summarized
signal can be reliably and accurately extracted before and after in Table III for different window lengths. We also compared
the change in position. During the position change the system the performance of the camera with these of the individual

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 VAN GASTEL et al.: CAMERA-BASED VITAL SIGNS MONITORING DURING SLEEP – A PROOF OF CONCEPT STUDY 1413

TABLE III
RESULTS OF THE CAMERA-DERIVED RESPIRATORY SIGNAL FOR DIFFERENT LENGTHS OF THE ANALYSIS WINDOW

Fig. 12. Examples of the respiration signals for the scenarios: (left)
stable, (middle) variations in breathing rate, and (right) movements.
Fig. 10. The percentage of time (PERC) where the respiration mea- All examples are segments of 5 minutes and computed with analysis
surement error is within the range 1–10 BRPM, grouped per sleep stage windows of 20 s.
and position. All measurements use the median of the breathing rates
from the PSG respiratory sensors as reference.
TABLE IV
BLOOD OXYGEN SATURATION CHARACTERISTICS OF THE DATASET. THE ODI
VALUES ARE CALCULATED BY THE PSG SOFTWARE

C. Oxygen Saturation
The dataset contains blood oxygen saturation values in the
Fig. 11. The duration distributions of unreliable segments for different range 83–99%, as the measurement corresponding with the
accuracy criteria, where the integer values indicates the respiration rate lowest saturation value of patient 4 is likely corrupted as all
error in breaths per minute (BRPM). The inset is a zoomed version of neighboring values are close to the average saturation value. An
the lower part of the graph.
overview of the oxygenation values is provided by Table IV,
where ODI ≤ 3% and ODI ≤ 4% represent the Oxygen De-
respiratory PSG sensors. The results of this comparison for a 10 s saturation Index (ODI), defined as the number of times per
window length are displayed in Fig. 10. Additionally, the dura- hour of sleep that the blood’s oxygen level drop by a certain
tions of unreliable segments for different accuracy criteria are percentage from the baseline. From recent literature, screening
displayed in Fig. 11. Fig. 12 displays examples of the respiratory with oximetry may be indicated for the detection of OSA in
signal for different scenarios. The first example shows a scenario select patients [16]. However, further study is needed before a
where the subject is stable and breathing at a quasi-constant rate. definitive recommendation can be made. For evaluation met-
The second example shows a scenario with large variations in rics, we calculated the mean-absolute-error (MAE), root-mean-
breathing rate in the range 12–25 breaths/min. The last example squared-error (RMSE), bias (B) and the percentage (PERC)
is a scenario where there is significant body motion during the where the measurements are within the clinically acceptable
second part of the segment. accuracy criterion specified in the International Standard for

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 1414 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 25, NO. 5, MAY 2021

TABLE V
RESULTS OF THE CAMERA-DERIVED BLOOD OXYGEN SATURATION
ESTIMATES

Fig. 15. The calculated ODIs in the range 3–10% for the camera and
finger-oximeter. The performance has been evaluated with a moving
median filter with lengths up to 30 s.

baseline value. The second column contains two examples of

consecutive desaturations within the 5 minutes segments, typ-
ical for apneic events. The last column contains examples of
estimated saturation values during body movements: changes in
sleep position (E) and severe movements (F).

D. Desaturation Events
Fig. 13. (Left) The percentage of time (PERC) where the measure-
ment error is within the range 1–10 pp, grouped per sleep stage and The evaluation results so far give a good impression about
position. (Right) The duration distributions of unreliable segments for
different accuracy criteria.
the coverage and how sleep position and sleep stage affects the
performance. Because for the majority of time the oxygenation
levels of the patients are stable, the performance during apneic
events is masked. In order to be useful for apnea detection, it is
important that the camera can detect these events. We therefore
computed all ODIs in the range 3–10% for both the camera and
the finger-oximeter, and evaluated a range of filter lengths. The
ODIs reported in Table IV are calculated by the PSG device
and do therefore not allow a direct comparison with the camera
ODIs. According to the American Academy of Sleep Medicine
(AASM) guidelines, an ODI event is scored during a “≤ x%
oxygen desaturation from a pre-event baseline”, where x is
typically 3 and/or 4. The definition of the pre-event baseline is
however not specified in the guidelines. We used a 30 s moving
median filter as a reasonable trade-off between the tracking of
Fig. 14. Examples of oxygen saturation values for the scenarios: (A+B)
stable with different baseline values from P2 and P3, (C+D) repetitive baseline trends and the immunity to apnea-induced baseline
desaturations during apneic events from P1 and P6, (E) changes in variations, i.e. desaturations. The calculated ODI values of the
sleep positions from P5, and (F) severe body movements from P6. All entire dataset are displayed in Fig. 15, the results per patient are
examples are segments of 5 minutes.
displayed in Fig. 19.

pulse-oximeter manufacture ISO 80601-2-61-2011 [17]. This IV. DISCUSSION

criterion requires an accuracy of ≤ 4% error in the oxygen The main focus of this study is to verify if the camera-derived
saturation range 70–100%, i.e. for all values in the dataset, when physiological signals are reliable surrogate measurements for a
compared to values obtained by blood gas analysis. We did subset of the PSG sensors. The results show that although the
not use filtering or any other type of post-processing. As the physiological information extracted from the related PSG sen-
camera estimates oxygenation levels from the face, whereas the sors is more accurate, the camera-derived parameters correlate
finger-oximeter is placed much more distal from the heart, and as fairly well with its references. Also, there are conditions where
the finger probes have their own internal delay, an empirically the PSG sensors provide inaccurate measurements, e.g. due to
selected delay of 8 s was added to the camera estimates. The sensor misplacement/displacement or movements. We observed
results are listed in Table V, the durations of the unreliable that there are situations where the camera provides more reli-
measurements for different accuracy criteria are presented in able measurements when compared to its PSG counterpart(s)
Fig. 13. Examples of oxygen saturation values obtained for and vice versa. Movements disorders during sleep often affect
different scenarios are displayed in Fig. 14. The first column the limbs, e.g. the periodic limb movement disorder (PLMD)
contains examples of stable oxygenation values with a different characterized by twitching, flexing, and jerking movements of

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 VAN GASTEL et al.: CAMERA-BASED VITAL SIGNS MONITORING DURING SLEEP – A PROOF OF CONCEPT STUDY 1415

Fig. 18. The histograms of the variance in oxygen saturation for the
different sleep position indicate the occurrence of apneic events to be
predominantly in supine position.

Fig. 16. Comparison of the camera-derived pulse signal with the syn-
chronized PPG and ECG PSG data for: (A) a scenario where the finger-
probe is more affected by artifacts compared to the camera, (B) an event the results. For ROI sizes < 0.015 the performance rapidly
with severe motion where both camera and PSG PPG measurements
are corrupted, and (C) a scenario with a weak, irregular pulse signal
decreases, indicating the lower bound.
and limited skin visibility. For the analyses of the respiration results, we compared the
performance of the camera with the individual respiratory PSG
sensors. The results show that, similar to pulse, the best perfor-
mance is achieved for the deeper sleep stages. The sleep position
has an effect on the performance, where both side positions show
better performance compared to the back. Although the exact
cause is unclear, this discrepancy has likely to be explained
by the position-dependent occurrence of apneic events, since
Fig. 17. Analysis on the effects of the ROI size on the performance of
it cannot be related to the ROI size, which is largest for the
the camera-derived signals. The ROI size is expressed as the fraction of sleep position on the back. The supine sleep posture produces a
the total frame size. Sample size is the number of processing windows. detrimental effect on breathing function as has been studied by
Oksenberg et al. [19]. This body position worsens the breathing
the legs and arms during non-REM sleep. Since the camera function of the majority of OSA patients and its associated
predominantly captures the face due to sheet covering, it is much body movements could therefore be an explanation for the
less susceptible to these movements when compared to sensors observation. The discrepancy between supine and lateral is also
attached to extremities, e.g. the finger-probe. An example of such larger for respiration compared to pulse, as can be observed from
a scenario is given by the third column of Fig. 9. A comparison of Fig. 7 and Fig. 10. To gain further evidence for the hypothesis
the camera-derived pulse signal and the PPG and ECG sensory that body movements associated with apneic events explain
data from the PSG for a similar scenario is displayed in Fig. 16A. the lower performance, we computed the histograms of the
When comparing the pulse results for the different sleep variances in oxygen saturation for the three sleep positions,
stages, it can be observed that the deeper sleep stages provide visualized in Fig. 18. Larger variations in oxygen saturation
the best results. Body movements could be an explanation for indicate apneic events. As can be observed from the figure,
this, as the movement characteristics differ between sleep stages apneic events predominantly occur in the supine (back) sleep
as has been studied by Wilde-Frenz et al. [18]. There is a strong position, with an average variation of [0.32 1.04 0.37] for the
relationship between rate of body movements and sleep stages, right, back and left sleep position, respectively. The sleep stage
with the rate decreasing along the following sequence of stages: and position dependency cannot be observed for the respiratory
Wake > N1 > REM > N2 > N3. Body position does not PSG sensors. Only during wake all sensors show a degradation
have a large impact on the performance, although both sleeping in performance, which is likely caused by motion artifacts
positions on the side provide better results than the back. This combined with a small sample size.
is encouraging as the areas of the face which have the strongest The sleep and position dependency for oxygen saturation
signal, forehead and cheeks, are largely occluded for the camera show a similar trend as compared to pulse and respiration,
during side sleep. The size of the ROI does not seem to be an again likely caused by body movements, either because of
explantation for this observation, as can be seen from Fig. 6. apneic events or restlessness. The validation of the estimated
Apart from patient 2, the ROI of all patients laying on the back oxygen saturation levels is performed with the finger-oximeter
is on average larger than for the other positions. To investigate PSG sensor. Both because of its anatomical location and the
the influence of the ROI size on the performance, we analysed the unavailability of un-post-processed data, this is not an ideal
quality of the pulse signal as function of the ROI size. In Fig. 17 reference. Forehead and ear probes are closer to the heart and
we plotted the signal-to-noise (SNR) and mean-absolute-error therefore respond more quickly than distal extremity probes; an
(MAE) as function of the ROI size, expressed as the fraction of average delay of 15 seconds between ear and finger has been
the total frame size, and added the sample size per ROI size. It can measured [20]. The response difference compared to central
be observed that the performance is almost invariant for a ROI SaO2 is also compounded by hypoxemia and slower peripheral
size in the range 0.015 − 0.1. For larger ROI sizes, the sample circulation such as low cardiac output states. As such, forehead
size becomes very small, which affects the interpretability of reflectance probes are often preferred in critically ill patients.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 1416 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 25, NO. 5, MAY 2021

Fig. 19. The calculated ODIs of all patients in the range 3–10% for the camera and finger-oximeter. The performance has been evaluated with a
moving median filter with lengths up to 30 s.

All of these response delays become clinically more important TABLE VI

OVERVIEW OF TREATMENT AFTER SLEEP STUDY
during rapid desaturation, such as those present in our dataset.
To verify the feasibility of the video system for the diagnosis
of OSA, we calculated the ODI values for the camera the
PSG finger probe. When comparing the ODI values for both
sensors as visualized in Fig. 15 and Fig. 19, it can be observed
that the camera-derived ODI values are consistently higher.
Especially for the ODIs associated with large desaturations,
increasing the filter lengths leads to a better correlation between
camera and finger probe. For the ODIs associated with smaller
desaturations, the typically used 3 and 4%, correlation improves
but only for the longest evaluated filter lengths. This can be
explained by the inherently noisier estimate of the camera,
where random fluctuations around the baseline value are
46.5 hours. Results are analyzed grouped by sleep stage and
classified as desaturation event.
sleep position. The camera can detect pulse and respiratory rate
Limited skin visibility is a factor which leads to a decreased
within 2 BPM accuracy 92% and 91% of the time, respectively.
performance as can be observed from our analysis in Fig. 17, and
Estimated blood oxygen saturation are within 4 percentage-
exemplified e.g. by patient 8 in Fig. 16C. When little skin is visi-
points of the finger-oximeter 89% of the time. We consider
ble due to occlusion, e.g. sensors covering the skin, head position
these results encouraging and demonstrate the potential of using
and/or sheets/pillow, possibly combined with a weak pulse sig-
remote PPG in NIR for sleep monitoring. Remaining challenges
nal, the grouping of regions with skin-pixels to create the ROI is
are head movements and limited skin visibility, both affecting
difficult. A possible solution would be to install cameras at mul-
the ROI selection. In order to use the system for the screening
tiple viewpoints for better coverage. This comes however with
and diagnosis of sleep apnea, improvements on the setup and/or
increased costs and practical difficulties, e.g. space limitations
front-end are required to make it more tolerant to movements
and illumination. For monitoring of respiration the limited skin
associated with apneic events.
visibility could be overcome by extracting different respiratory-
affected features from the camera data, e.g. by adding motion-
based respiration detection [21]. However, the damping of the APPENDIX
torso movements by the sheets would likely affect the perfor- ADDITIONAL RESULTS
mance. Furthermore it requires a larger viewing angle, whereas In this Appendix additional results on oxygen saturation,
we limited our viewing angle to the head end only. AHIs, treatment and metadata of the study dataset are presented.

A. Oxygen Saturation
V. CONCLUSION
The ODIs values of each patient calculated from the oxygen
We presented the results of a camera-based solution for the
saturation results are visualized in Fig. 19.
monitoring of parameters which are relevant for the diagnosis of
sleep apnea. The physiological parameters pulse rate, respiration
and blood oxygen saturation have been continuously monitored B. AHI and Treatment
during the nocturnal sleep registration at a specialized sleep The Apnea Hypopnea Indices of the patients in the dataset are
institute using remote photoplethysmography. Data of eight [52.7, 35.4, 10.8, 7.7, 9.4, 46.7, 11.7, 18.6] for Patient (P) 1–8,
patients enrolled for a full PSG have been recorded resulting respectively. The treatment of the patients after the sleep study
in a dataset consisting of video data with a total duration of are listed in Table VI.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 VAN GASTEL et al.: CAMERA-BASED VITAL SIGNS MONITORING DURING SLEEP – A PROOF OF CONCEPT STUDY 1417

Fig. 20. Hypnograms (left) and body positions (right) of the eight recorded patients.

TABLE VII
SLEEP STAGE CHARACTERISTICS OF THE DATASET. D DENOTES THE DURATION IN MINUTES AND P DENOTES THE PERCENTAGES FOR TIME IN BED (TIB)
AND TOTAL SLEEP TIME (TST), RESPECTIVELY

C. Metadata [3] S. Patel, L. Ruoff, T. Ahmed, T. Unadkat, and J. Lee, “Validation of basis
science advanced sleep analysis,” Basis Sci., 2014. Accessed: Apr. 6,
Information about the sleep registrations of each patient are 2020. [Online]. Available: https://nanopdf.com/download/validation-of-
visualized in Fig. 20 and Table VII, where the Non-REM sleep basis-science-advanced-sleep-analysis_pdf
[4] M.-C. Yu, H. Wu, J.-L. Liou, M.-S. Lee, and Y.-P. Hung, “Multiparameter
stages are abbreviated with N. sleep monitoring using a depth camera,” in Proc. Int. Joint Conf. Biomed.
Eng. Syst. Technol., 2012, pp. 311–325.
ACKNOWLEDGMENT [5] M. Kagawa, K. Ueki, H. Tojima, and T. Matsui, “Noncontact screening sys-
tem with two microwave radars for the diagnosis of sleep apnea-hypopnea
The authors would like to thank L. van den Heuvel from syndrome,” in Proc. IEEE 35th Annu. Int. Conf. Eng. Med. Biol. Soc.,
Philips Research, and B. Hoondert and P. Bijkerk from 2013, pp. 2052–2055.
Kempenhaeghe Center for Sleep Medicine for their help in the [6] M. Martinez and R. Stiefelhagen, “Breath rate monitoring during sleep
using near-ir imagery and pca,” in Proc. IEEE 21st Int. Conf. Pattern
preparation and execution of the study. Furthermore, we thank Recognit., 2012, pp. 3472–3475.
all patients from Kempenhaeghe for their participation. [7] J. H. Shin, Y. J. Chee, D.-U. Jeong, and K. S. Park, “Nonconstrained sleep
monitoring system and algorithms using air-mattress with balancing tube
REFERENCES method,” IEEE Trans. Inf. Technol. Biomed., vol. 14, no. 1, pp. 147–156,
Jan. 2010.
[1] C. Li, E. S. Ford, G. Zhao, J. B. Croft, L. S. Balluz, and A. H. Mokdad, [8] M. van Gastel, S. Stuijk, and G. de Haan, “Motion robust remote-ppg
“Prevalence of self-reported clinically diagnosed sleep apnea according to in infrared,” IEEE Trans. Biomed. Eng., vol. 62, no. 5, pp. 1425–1433,
obesity status in men and women: National health and nutrition examina- May. 2015.
tion survey, 2005–2006,” Prev. Med., vol. 51, no. 1, pp. 18–23, 2010. [9] M. Van Gastel, S. Stuijk, and G. De Haan, “Robust respiration detection
[2] T. Young, M. Palta, J. Dempsey, J. Skatrud, S. Weber, and S. Badr, “The from remote photoplethysmography,” Biomed. Opt. Express, vol. 7, no. 12,
occurrence of sleep-disordered breathing among middle-aged adults,” New pp. 4941–4957, 2016.
Engl. J. Med., vol. 328, no. 17, pp. 1230–1235, 1993.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.
 1418 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 25, NO. 5, MAY 2021

[10] M. Van Gastel, S. Stuijk, and G. De Haan, “New principle for measuring [16] N. H. Rashid, S. Zaghi, M. Scapuccin, M. Camacho, V. Certal, and
arterial blood oxygenation, enabling motion-robust remote monitoring,” R. Capasso, “The value of oxygen desaturation index for diagnosing
Sci. Rep., vol. 6, no. 1, pp. 1–16, 2016. obstructive sleep apnea: A systematic review,” Laryngoscope, 2020,
[11] T. Vogels, M. van Gastel, W. Wang, and G. de Haan, “Fully-automatic doi: 10.1002/lary.28663.
camera-based pulse-oximetry during sleep,” in Proc. IEEE Conf. Comput. [17] “Medical electrical equipment – Part 2-61: Particular requirements for
Vis. Pattern Recognit. Workshops, 2018, pp. 1349–1357. basic safety and essential performance of pulse oximeter equipment,”
[12] M. van Gastel, W. Verkruysse, and G. de Haan, “Data-driven calibration International Organization for Standardization, Geneva, CH, Standard,
estimation for robust remote pulse-oximetry,” Appl. Sci., vol. 9, no. 18, Feb. 2018.
2019. [Online]. Available: https://doi.org/10.3390/app9183857 [18] J. Wilde-Frenz and H. Schulz, “Rate and distribution of body move-
[13] G. Matar, J.-M. Lina, J. Carrier, and G. Kaddoum, “Unobtrusive sleep mon- ments during sleep in humans,” Perceptual Motor Skills, vol. 56, no. 1,
itoring using cardiac, breathing and movements activities: An exhaustive pp. 275–283, 1983.
review,” IEEE Access, vol. 6, pp. 45 129–45 152, 2018, doi: 10.1109/AC- [19] A. Oksenberg and D. S. Silverberg, “The effect of body posture on
CESS.2018.2865487. sleep-related breathing disorders: Facts and therapeutic implications,”
[14] R. B. Berry et al., The AASM Manual for the Scoring of Sleep and Sleep Med. Rev., vol. 2, no. 3, pp. 139–162, 1998.
Associated Events: Rules, Terminology and Technical Specifications, [20] P. Lindholm, S. L. Blogg, and M. Gennser, “Pulse oximetry to detect
vol. 1, Westchester, IL, USA: AASM Amer. Acad. SLEEP Med., hypoxemia during apnea: Comparison of finger and ear probes,” Aviation,
2012. Space, Environ. Med., vol. 78, no. 8, pp. 770–773, 2007.
[15] G. De Haan and V. Jeanne, “Robust pulse rate from chrominance- [21] M. Bartula, T. Tigges, and J. Muehlsteff, “Camera-based system for
based rppg,” IEEE Trans. Biomed. Eng., vol. 60, no. 10, pp. 2878–2886, contactless monitoring of respiration,” in Proc. IEEE 35th Annu. Int. Conf.
Oct. 2013. Eng. Med. Biol. Soc., 2013, pp. 2672–2675.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on April 19,2023 at 20:14:00 UTC from IEEE Xplore. Restrictions apply.