The SWELL Knowledge Work Dataset for Stress and User

Modeling Research

Saskia Koldijk Maya Sappelli Suzan Verberne

ICIS, Radboud University ICIS, Radboud University ICIS, Radboud University, The
& TNO, The Netherlands. & TNO, The Netherlands. Netherlands.
[email protected] [email protected] [email protected]
Mark A. Neerincx Wessel Kraaij
II, Delft University of ICIS, Radboud University
Technology & TNO, The Netherlands.
& TNO, The Netherlands. [email protected]
[email protected]

ABSTRACT 1. INTRODUCTION
This paper describes the new multimodal SWELL knowl- Nowadays most work involves computer usage and infor-
edge work (SWELL-KW) dataset for research on stress and mation processing. People that use and produce information
user modeling. The dataset was collected in an experiment, as their main task are called knowledge workers. They typi-
in which 25 people performed typical knowledge work (writ- cally experience all sorts of demands during their work days,
ing reports, making presentations, reading e-mail, searching such as several tasks that need to be finished before a dead-
for information). We manipulated their working conditions line (high workload, temporal demand). For this they need
with the stressors: email interruptions and time pressure. to combine different information sources, for example from
A varied set of data was recorded: computer logging, facial the internet (requiring mental effort). Incoming emails may
expression from camera recordings, body postures from a be an important source of distraction during a task (poten-
Kinect 3D sensor and heart rate (variability) and skin con- tially causing frustration). In case people feel they cannot
ductance from body sensors. The dataset made available not handle the demands posed upon them, they can experience
only contains raw data, but also preprocessed data and ex- stress [3]. Stress is a broad concept referring to psychologi-
tracted features. The participants’ subjective experience on cal and biological processes during emotional and cognitive
task load, mental effort, emotion and perceived stress was demanding situations. We follow a pragmatic approach and
assessed with validated questionnaires as a ground truth. define stress in terms of: (1) the task load, which poses de-
The resulting dataset on working behavior and affect is a mands on the worker, (2) the mental effort, which the worker
valuable contribution to several research fields, such as work needs to handle a task and (3) the emotional response to a
psychology, user modeling and context aware systems. task. In this paper, we focus on short term effects of stres-
sors that can be measured within a 3 hour work session.
Stress is a well-known experience in our connected envi-
Categories and Subject Descriptors ronments. Ruff [16] speaks of ‘hurry sickness’ as “the be-
E.0 [Data]: General; H.1.2 [Models and Principles]: User/ lief that one must constantly rush to keep pace with time”
Machine Systems—Human factors, Human information pro- and ‘plugged in compulsion’ as “the strong need to check
cessing; J.4 [Social and Behavioral Sciences]: Psychol- mail and the internet to stay in touch”. Mark, Gudith and
ogy Klocke [13] investigated the cost of interruptions and came
to the conclusion that “after only 20 minutes of interrupted
General Terms performance people reported significantly higher stress, frus-
tration, workload, effort and pressure”. Stress from time to
Human Factors, Experimentation, Measurement time with enough room for recovery is no problem [2]. How-
ever, when stress builds up this can be a danger to well-
Keywords being, in the worst case resulting in burn-out.
Dataset; stress; mental state; facial expressions; body pos- In our project SWELL (Smart Reasoning for Well-being
tures; computer interaction; physiology at Home and at Work)1 we aim to develop ICT tools that
help knowledge workers to cope with stress and gain more
Permission to make digital or hard copies of all or part of this work for personal or well-being at work [8]. We want to interpret recordings in
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full cita-
the office real-time in terms of stress and the context in
tion on the first page. Copyrights for components of this work owned by others than which it appears. Based upon this information, we aim to
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- develop coaching software that can help knowledge work-
publish, to post on servers or to redistribute to lists, requires prior specific permission ers to gain a more healthy work style. Moreover, we want
and/or a fee. Request permissions from [email protected]. to develop smart information support tools that assist the
ICMI ’14, November 12 - 16 2014, Istanbul, Turkey knowledge worker in handling the large amount of (incom-
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-2885-2/14/11 ...$15.00. 1
http://dx.doi.org/10.1145/2663204.2663257. http://www.swell-project.net

291
ing) information he has to work with. In this way, we extend This paper is structured as follows. We first present some
traditional approaches (e.g. questionnaires or department related work (Section 2). We then outline our experimental
wide interventions, [11, 12]) by empowering individual users setup in which the dataset was collected (Section 3). We
to self-manage their own well-being. describe the dataset in detail in Section 4. Some example
To be able to develop the ICT tools we envision, research analyses are presented in Section 5. We finish with a Dis-
communities like work psychology, user modeling and con- cussion and Conclusion (Section 6 and 7).
text aware systems are in need of a good dataset. This
dataset should ideally have the following characteristics: Data 2. RELATED WORK
should be recorded in a realistic office setting. Stressors
In this section we present some related research on work
should be manipulated in a systematic way and subjective
psychology and user modeling, in which sensor data is used
experience should be assessed with validated questionnaires,
to estimate stress, mental or affective states. We specifically
to be able to investigate the effects of stressors. A multi-
address the type of sensors used, the context in which data
modal set of sensors from different research fields should be
has been collected and the kind of inferences that have been
used, to enable multidisciplinary research. The focus should
made. This gives a theoretical framework for research on
lie on sensors that are readily available in office settings, to
our collected dataset.
make the to be developed system usable outside the lab. To
In work psychology, questionnaires are commonly used to
our knowledge no such dataset existed.
get insight in the general working experiences (e.g. [20]).
In this paper we present a newly collected rich dataset
Advances in sensing, as well as the quantified self movement
which has these characteristics. Our dataset overcomes three
make it possible to extend such an approach with on-site
drawbacks that are typically observed in related work:
measurements.
• Instead of a rather artificial task, participants perform Work in the area of affective computing investigates the
natural office work with systematically manipulated possibility of inferring stress and emotion from sensor data.
stressors. Most often, physiological sensors are used and data are col-
lected in experimental environments. In research by Riera
• Instead of expensive and/or obtrusive equipment, we et al. [15], for example, electroencephalography (EEG) and
decided to combine a variety of sensors that can easily facial electromyography (EMG) data were collected. The
be deployed in real-world office settings. authors show that EEG and EMG can be used for moni-
• Instead of only collecting data for our own use, we toring emotion (valence and arousal) and stress. Although
will make the anonymized dataset available for access its great potential, we think deploying EEG in a daily office
by the scientific community, for benchmarking of tech- setting is not yet realistic. Other common measurements
niques and algorithms. We will not only provide raw in stress research are pupil diameter and electrocardiogram
data, but also data in preprocessed and interpreted (ECG). Mokhayeri, Akbarzadeh-T and Toosizadeh [14], for
form. example, collected such data in context of the Stroop color-
word (SCW) test. They state that pupil diameter and ECG
With our new dataset, we aim to bring research on psy- have great potential for stress detection. However, the ques-
chology and computer science together. With this dataset, tion that arises is: can we also make an estimate of affective
research questions from several fields can be answered, for and mental states outside the lab? We see some potential
example: for ECG measurements, with the rise of wearable sensors,
• Work psychology: What effect do stressors like time which are becoming more and more integrated into devices
pressure have on the working behavior of knowledge as watches and bracelets. But, besides measuring the phys-
workers? What is the effect of an incoming email? iological stress response directly, we also see great potential
What effect do stressors have on subjective experience in measuring outward characteristics, such as facial expres-
of task load, mental effort, emotion or perceived stress? sions, postures or computer interactions as indicators for the
What is the relation between what people mean when user’s state.
they feel ‘stressed’ and the concepts of arousal, mental In related work, facial expressions are widely used for in-
effort and valence? Do we see effects of stressors in ferring emotions. The data are often recorded while emo-
physiological sensor data? tions are induced in participants. The publicly available
multimodal dataset described by Soleymani et al. [18], for
• User modeling: Can we estimate the mental state or example, was collected in context of watching emotion in-
emotion of knowledge workers from unobtrusive sensor ducing video clips and consists of: face videos, audio sig-
data? Do knowledge workers show particular affective nals, eye gaze data and physiological signals (EEG, ECG,
expressions during computer work? Are there typi- GSR, respiration amplitude, skin temperature). Although
cal facial expressions or postures that are indicative of this dataset is very interesting, emotions in a daily computer
mental effort, high workload or stress? work context are probably less intense than the valence or
arousal experienced during watching a movie clip. An inter-
• Context aware systems: Can we automatically deter-
esting question is whether people show facial emotions dur-
mine the task or topic someone is working on? Is there
ing computer work, and whether their facial expressions are
a relation between stress and the context in which it
indicative of mental states. Preliminary results by Dinges et
occurs? Can we filter irrelevant emails? Can we make
al. [4] suggest that high and low stressor situations could be
information retrieval more context aware?
discriminated based on facial activity in mouth and eyebrow
In this paper we mainly focus on work stress and user regions.
modeling. For more details about the data regarding context Regarding postures, Kapoor and Picard [7] present re-
recognition and information support, we refer to [17]. search in which posture data was collected together with

292
facial expressions and computer information while children an uninfluenced baseline of normal working. The order of
solved an educational computer puzzle. Sensors in the chair the two stressor conditions was counterbalanced, see Figure
were used to extract posture features (like leaning back, sit- 1. The within-subject design included relaxation breaks to
ting upright) and activity level (low, medium, high). Posture start each condition in a well-rested state.
information yielded the highest unimodal accuracy (80%) for
estimating interest (vs. uninterest). Performance was fur-
ther improved by adding facial expression and computer in-
formation. We conclude that posture information and move-
ment are an interesting source for estimating the users’ men-
tal state. We see potential for posture measurements in the
office, as with the Kinect recently an affordable 3D camera
with skeleton detection has entered the market. Figure 1: Design. For 13 participants order A was
Finally, in some research, stress or emotions are estimated used, for 12 participants order B.
from computer interaction data. Vizer, Zhou and Sears [19],
for example, investigated the effect of stress on typing pat-
terns. Participants first performed a mentally or physically
stressful task (e.g. remembering digits or exercising) and 3.2 Tasks
were then asked to write an email. Results indicate that The participants performed knowledge worker tasks on
stress can produce changes in typing patterns. This makes a desktop computer in a controlled lab setting. We asked
computer logging a valuable sensor for user state modeling. them to write reports and make presentations on predefined
We think not only typing patterns, but also more general topics (in English). We selected 6 topics on which people
computer behavior might be indicative of mental states, like with various backgrounds could work:
the amount of window switching, number of typos or time
• 3 opinion topics: Experience and opinion about ‘stress
spent browsing.
at work’, ‘healthy living’ and ‘privacy on the internet’.
Besides inferring stress or particular mental or affective
states, the context in which they appear can be interesting. • 3 information topics: ‘describe 5 Tourist attractions in
Computer interactions give rich insight in the user’s current Perth (West Australia)’, ‘plan a coast to coast road-
working behavior. Research by Koldijk et al. [10] shows trip in the USA’ and ‘write about the life of Napoleon’.
that it is possible to infer the task someone is working on
Some detail on what to include in the report was also given.
from computer interaction data. Moreover, one could add
Participants’ were allowed to look for information on the
analysis of contents worked on.
internet and use documents that we previously stored on
To conclude, research from various related fields shows the
the computer. This setting is typical for knowledge work
potential of using sensors for estimating stress, mental and
as available information can be combined with the worker’s
affective states and the context in which they appear. In
own input in a coherent way, with the purpose of generat-
each field, a particular setup of sensors is used. We decided
ing a new information product. During the task the email
to combine several of these in our unique dataset: computer
program Outlook was running and participants were told
interactions, video for facial expressions, Kinect 3D for pos-
to make use of information from incoming emails and reply
tures and body sensors for heart rate and skin conductance.
when necessary. In this way, a realistic office work scenario
was created.
3. DATA COLLECTION CONTEXT We wanted to ensure that the participants worked on the
In this section we present the experimental setup that was tasks seriously. Therefore we told them that it was impor-
used to collect data. tant to finish all required tasks for receiving the full subject
fee. Moreover we told them that they would have to give
3.1 Design one of the prepared presentations. After the experiment,
In our experiment we manipulated the conditions under we debriefed all participants and informed them that they
which our participants worked: did not need to give a presentation and would get the full
• Neutral: the participant was allowed to work on the subject fee.
tasks as long as he/she needed. After a maximum of 3.3 Procedure
45 minutes the participant was asked to stop and told
that enough data of ‘normal working’ was collected. To be able to record stress responses as a result of our
experimental manipulations, we instructed the participants
• Stressor ‘Time pressure’: the time to finish all tasks to not smoke or drink caffeine 3 hours prior to the experi-
was 2/3 of the time the participant needed in the neu- ment, as these are possible confounders. Before the experi-
tral condition (and maximally 30 minutes). ment started, the experiment and recordings were explained
and all participants signed a consent form to confirm that
• Stressor ‘Interruptions’: 8 emails were sent to the par-
the recorded data may be used for research purposes. Body
ticipant during the task. Some were relevant to one of
sensors were applied and while the experimenter checked the
the tasks, others were irrelevant. Some emails required
recordings, the participant read the experiment instructions
a reply, others did not. Examples are: “Could you look
and filled in a general questionnaire.
up when Einstein was born?” or “I found this website
The experiment was divided into three blocks for the dif-
with lots of nice pictures for presentations.”.
ferent stressor conditions, each taking approximately one
All participants worked under all 3 conditions. The neutral hour. Each of the experimental blocks started with a relax-
condition was always the first condition, in order to collect ation phase of about 8 minutes (which is typical for stress

293
research) in which a nature film clip was shown. Then the blame external factors. This might be of influence on par-
participants received instructions on the tasks to work on. ticipants’ stress perception or behavior. Moreover, partici-
In each block the participants were provided with 2 of the 6 pants were asked to rate their interest in the topics, as well
topics, which were randomly selected from the list, in such a as how difficult they found it to write a report or make a
way that always an opinion topic was combined with an in- presentation on a topic on a 7-point Likert scale (from ‘not
formation topic. The participants were instructed to write interesting / difficult’ to ‘very interesting / difficult’).
2 reports, one on each topic, and make 1 presentation on
one of the topics (participants could choose the topic). To 3.6 Sensors
prevent learning effects, the participants were provided with Computer logging. Computer interactions were logged
different topics in every block. In both stressor conditions, with the key-logging application uLog (version 3.2.5, by Noldus
participants were provided a count-down clock for showing Information Technology), which ran as a background appli-
them the remaining time. cation on the users’ computer.
After completion of the tasks, the participants were asked Video. Video recordings of the participants face and up-
to fill in a questionnaire about the current block. This pro- per body were made with a high-resolution USB camera
cedure of relaxation, tasks execution and questionnaire was (iDS uEye UI-1490RE, 1152x768) which was positioned be-
then repeated for block 2 and 3 (see Figure 1). Between the low the participants’ monitor. The AVI files from the USB
conditions the subjects were allowed a short break and the camera were further analyzed using the facial expression
total experiment took about 3 hours. After the experiment analysis software FaceReader (version 5.0.7 RC 4.5 (Beta)).
the participants were debriefed. An additional webcam (Philips SPC 900NC, SVGA resolu-
tion) was placed above the participants’ monitor.
3.4 Apparatus Kinect 3D. The participants body posture was recorded
Participants performed their tasks on a computer (Dell with a Kinect (for Windows, model 1517) depth camera.
Latitude E6400) with Windows 7 Professional with a 17 The camera was placed in front of the participants at a
inch screen and mouse and keyboard (see Figure 2). Office distance of about 2 meters, such that their whole body,
2010 was installed, which the participants used for email including their legs under the desk were visible (see Fig-
(Outlook), report writing (Word) and making presentations ure 2). Besides 3D depth video, Kinect also recorded nor-
(Powerpoint). As a browser, Internet Explorer was used mal RGB video. Recordings were made with Kinect Stu-
with Google as default search engine. The start page of dio (v1.7.0), which resulted in xed-files. From the recorded
Internet Explorer was www.google.nl. Kinect data the depth image and information on the skele-
tal model were extracted using the Windows Kinect SDK
(v1.7). We smoothened the data with several predefined
filters.
Body sensors. ECG was recorded using a Mobi device
(TMSI) with self-adhesive electrodes. The electrodes were
placed across the heart, one below the participants right
collar-bone, the other left below the chest, with a ground-
ing electrode below the left collar-bone. Some preprocessing
was programed into the recording software Portilab2. To
record skin conductance, Mobi was used with finger elec-
trodes. These were fixed with Velcro tape around the lower
part of the thumb and ring finger of the participant’s non-
dominant hand. Recording frequency was 2048 Hz. All sig-
nals (ECG and skin conductance, raw and preprocessed)
Figure 2: Experimental set-up. were stored together in S00-files.
Additional Lab Recordings. The lab’s ceiling camera
and microphones were used for making records of the lab
during the whole experiment, as well as a screen capture of
3.5 Subjective Ratings the participant’s screen. The video files are encoded in AVI-
To collect a ground truth of the subjective experience af- format with a codec specific to the labs recording software
ter each block, we used a combination of validated ques- (GeoVision’s CCS5). Audio is encoded in separate wav-files.
tionnaires. Task load (in terms of mental demand, physical
demand, temporal demand, effort, performance and frustra- 3.7 Participants
tion) was determined with the ‘NASA-Task Load Index’ [6]. 25 students participated in our experiment, of which 8
Mental effort was assessed with the ‘Rating Scale Mental were female and 17 male. The average age was 25 (stan-
Effort’ [21]. Emotion response (in terms of valence, arousal dard deviation 3.25). Most participants were native Dutch.
and dominance) was determined with the ‘Self-Assessment- They were interns from TNO and students from Delft Uni-
Manikin Scale’ [1]. Moreover, we asked participants to re- versity of Technology who were approached by advertising.
port their perceived stress on a visual analog scale from ‘not Since these interns and students are experienced in handling
stressed’ to ‘very stressed’ (10 point scale). (large amounts of) information for their courses, and often
Furthermore, we asked the participant’s to fill in the ‘In- use computers as their most important tool, they are as-
ternal Control Index’ questionnaire [5]. People with an in- sumed to be representative of knowledge workers. Addi-
ternal locus of control tend to praise or blame themselves, tionally, they are experienced with the knowledge worker
whereas people with an external locus tend to praise or tasks we have chosen: writing reports and preparing presen-

294
tations. The participants received a standard subject fee for
their participation in the experiment. Table 2: Computer interaction features (aggregated
To assess whether the participants worked on the tasks per minute).
seriously, we checked the quality of the written reports and Type Feature Description
presentations. As the quality was satisfactory, none of the Mouse MouseActivity Number of all MouseEvents
subjects needed to be excluded from the corpus. Of the par- LeftClicks Number of left clicks
ticipants, 2 were left handed and 8 wore glasses (which could MouseWheel Number of mouse wheel
be of concern for the software analyzing facial expressions). scrolling
Results of our pre-questionnaire showed that none of the par- Key- KeyStrokes Number of all KeyEvents
ticipants indicated to have a heart disease or take medicine board ShortcutKeys Number of shortcut keys
which could have influenced their heart rate. About half (Ctrl+ c/x/v/z/s/a; Shift+Tab)
the participants indicated that they were physically active DirectionKeys Number of direction keys (ar-
before the experiment as they came by bike. 4 participants row left/right/up/down)
indicated that they had experienced stress prior to the ex- Characters Number of characters (a-z)
periment. None of the participants smoked, drank caffeine CharactersRatio #characters devided by
#keyStrokes
or alcohol 3 hours prior to the experiment. The participants
ErrorKeys Number of error keys
scored on average 3.67 on on the internal control index (scale (Backspace, Delete, Ctrl+Z)
from 1 to 5, with higher scores indicating more internal con- ErrorKeyRatio #errorKeys devided by
trol; stdv = 0.29). (#characters + #spaces)
Appli- AppChanges Number of application changes
4. DATASET cations TabfocusChange Number of tab focus changes
In this section we present the public SWELL-KW dataset
in more detail. We collected data from the following sen-
sors: computer logging, video, Kinect 3D and body sensors. Facial expressions from video. We do not include the
Handling this data requires expertise in different fields. We fully recorded videos in our dataset to keep our participants
preprocessed this data to get an aggregation of computer anonymous. Instead, we provide data files with the analysis
interactions, extraction of facial expressions, postures, heart of facial activation. These were extracted from the video per
rates and skin conductance levels. We finally aggregated timeframe using the software FaceReader. The characteris-
this data into features per minute. For an overview of all tics that are included in the dataset are: quality, estimates
available data see Table 1. We now fist describe the avail- on the orientation of the head, some global features of the
able fully preprocessed and aggregated feature data. Then face like looking direction and the amount of activation in
we describe the available raw data and preprocessing. several facial action units. Moreover, FaceReader provides
an estimate of the subjects emotion, which is also available in
4.1 The Feature Data our dataset. We parsed these files to get a more intelligible
The feature dataset contains our completely preprocessed timestamped table format, which will also be made avail-
data, aggregated per minute, for all 25 participants. It con- able. Besides data per video frame, we also calculated aver-
tains the following features: 12 computer interaction fea- ages per minute for all characteristics (see Table 1), which
tures, 40 facial expression features, 88 body posture features are contained in the feature dataset.
and 3 physiology features as listed in the right column of Ta-
ble 1. The feature dataset is annotated with the conditions
under which the data was collected. Per participant three
times 6 minutes relaxation data are included, ca. 45 minutes
of working under normal conditions, ca. 45 minutes working
with email interruptions and ca. 30 minutes working under
time pressure.
Moreover, we provide the scores on our questionnaire items
as ground truth for the subjective experience in each condi-
tion, see Table 3. As 25 participants each rated 3 conditions,
this yields 75 ratings in total.

4.2 The Raw Data and Preprocessing

Besides the completely preprocessed and aggregated data,
we also provide some raw data and files resulting from our
preprocessing, as listed in the middle column of Table 1.
Computer logging. The computer logging software recorded Figure 3: Kinect joints and Kinect orientation from
detailed timestamped information in XML format about Kinect SDK. Images from: http://msdn.microsoft.com
each computer event. Examples of computer events are
mouse clicks, mouse scrolls and application changes. More-
over we parsed the files and printed them in a more intel- 2
http://www.noldus.com/human-behavior-research/
ligible timestamped table format, which will also be made products/ulog
available. Finally, we computed several relevant mouse, key- 3
http://www.noldus.com/human-behavior-research/
board and application characteristics per minute (listed in products/facereader
4
Table 2), which are contained in the feature dataset. http://www.tmsi.com/en/products/mobi.html

295
Table 1: Our dataset contains data from 25 participants (3 hours each). The listed raw and preprocessed
sensor data, as well as a feature dataset (aggregated per minute) will be made available.
Type Available raw and preprocessed data Available features (#features)
Computer interactions uLog output2 Mouse (3)
(i.e. xml-logs of all computer events) Keyboard (7)
Parsed data Applications (2)
(i.e. txt-file with timestamped data) (for details see Table 2)
Facial expressions FaceReader output3 Head orientation (3)
(i.e. txt-logs with facial information and emotions) Facial movements (10)
Parsed data Action Units (19)
(i.e. txt-file with timestamped data) Emotion (8)
Body postures Joint coordinates extracted with Kinect SDK Distance (1)
(i.e. txt-file with timestamped data) Joint angles (10)
Angles of the upper body Bone orientations (3x11))
(i.e. txt-file with timestamped data) (as well as stdv of
the above for amount of movement (44))
Physiology Data from Mobi4 Heart rate (variability) (2)
(i.e. S00-files with raw and filtered signals) Skin conductance (1)

Body postures from Kinect 3D sensor. We do not the distance between the found peaks (R-R). To determine
include the recorded 3D Kinect files in our dataset to keep the heart rate variability we took the root mean square of
our participants anonymous. Instead, we provide data files all these peak distances (RMSSD). Due to some remaining
with analysis of the participant’s body posture per time- noise in the signal, the peak finding algorithm sometimes
frame. These were extracted from the 3D Kinect recordings failed to accurately detect peaks. Therefore we excluded all
using the Kinect SDK. By fitting the Kinect skeletal model 1-minute time frames in which more than one peak distance
(see Figure 3, left), we got coordinates of all body joints per appeared unusual. We defined an unusual peak distance as
frame. This data will be made available. We further used a distance larger than 1.2 seconds where probably a peak
these joint coordinates to determine joint angles between was missed (or otherwise the HR would be below 50bpm) or
bones of the upper body, for example the angle between the a distance smaller than 0.5 seconds where probably an ex-
upper and lower arm. Moreover, we determined bone ori- tra peak was detected (or otherwise the HR would be over
entations of the upper body relative to the x, y and z axis 120bpm). The resulting heart rate and heart rate variability
(see Figure 3, right), for example the angle between the left are contained in the feature dataset.
shoulder and the up pointing y axis.5 This information on Moreover, we provide raw skin conductance data. We also
angles per frame will be made available. From the depth calculated the average skin conductance level by averaging
image the average distance of the user was also determined. the raw signal per minute, which is contained in the feature
Finally, we determined average angles per minute, which are dataset.
contained in the feature dataset. We also calculated stan-
dard deviations for each minute, to determine features that
indicate the amount of movement and changes in joint an- 5. EXAMPLE ANALYSES
gles. These are also contained in the feature dataset. In this section we present some research that was done
Physiology from body sensors. We provide raw and based on our dataset, as an example of its use.
preprocessed ECG data. The raw ECG signal was filtered Work stress. To find relations between the measured
as described in the TMSI6 manual: First a high pass filter concepts, we performed a correlation analysis on the ques-
(8Hz) was applied to filter out large fluctuations in the sig- tionnaire data. We found that perceived stress is moderately
nal. A 15ms second delay was added, together with a delta related to high task load in terms of mental demand, tem-
filter to let the low frequency parts of the signal disappear. poral demand and frustration. Moreover, stress is related to
To be independent of the direction of the QRS complex (due emotion in terms of negative valence and high arousal. For
to morphology of the ECG), we took the absolute signal. Fi- more details on these results, see [9].
nally, a moving window averager (0.1sec) was added to get To investigate the effect of our stressors on the partici-
the envelope of the signal. This yielded a filtered signal with pants’ subjective experience, we compared the questionnaire
clear peaks. The raw and preprocessed ECG data will be ratings of the neutral baseline condition with the time pres-
made available. sure and email interruption conditions (see Table 3). Under
We also calculated the heart rate and heart rate variabil- the stressor time pressure, participants experienced signif-
ity. Therefore, we processed the filtered data further in Mat- icantly higher temporal demand and higher arousal. The
lab. First of all we applied a peak detection algorithm to the stressor email interruptions yielded reports of more mental
filtered signal. To determine the heart rate, the found peaks effort, more positive valence and more dominance. We found
were counted per 1 minute time-frame. Then we calculated that perceived stress did not differ significantly between the
stressor and neutral conditions. Stress might be a too com-
5 plex concept to measure in a short-termed work task. For
We use a projection to the plane to distill only variance in
one direction. more details on our results, see [9]. These analyses show the
6
http://www.tmsi.com potential of using the dataset for research on the effect of

296
Table 3: Subjective experience data (one rating per block). Average values for the Neutral, Interruption and
Time pressure condition can be found in the last 3 columns.
Type Feature Description N I T
TaskLoad MentalDemand How much mental and perceptual activity was required (e.g. 4.9 5.4 4.9
(NASA-TLX) (0: low - 10: high) thinking, deciding, calculating, remembering, looking, search-
ing, etc.)?
PhysicalDemand How much physical activity was required (e.g. pushing, 1.9 2.3 2.7
(0: low - 10: high) pulling, turning, controlling, activating, etc.)?
TemporalDemand How much time pressure did you feel due to the rate or pace 5.7 5.9 7.1
(0: low - 10: high) at which the task or task elements occurred?
Effort How hard did you have to work (mentally and physically) to 5.2 5.9 6.1
(0: low - 10: high) accomplish your level of performance?
Performance How successful do you think you were in accomplishing the 4.8 6.1 6.0
(0: poor - 10: good) goals of the task set by the experimenter (or yourself)?
Frustration How insecure, discouraged, irritated, stressed and annoyed 3.5 3.6 3.5
(0: low - 10: high) versus secure, gratified, content, relaxed and complacent did
you feel during the task?
Mental Effort MentalEffort How high was the mental effort for the tasks you just finished? 5.5 6.5 6.3
(RSME) (0: no - 10: extreme effort)
Emotion Valence (1 - 9) How do you feel at this moment? (unhappy - happy) 4.8 5.7 5.3
(SAM) Arousal (1 - 9) How do you feel at this moment? (calm - excited) 3.3 3.9 4.6
Dominance (1 - 9) How do you feel at this moment? (submissive - dominant) 5.2 6.2 5.9
Stress Stress How stressed do you feel? 2.9 3.2 3.8
(VAS) (0: not - 10: very stressed)

work stressors on experience and behavior. In collecting and preprocessing the data, we encountered a
User modeling. We are aiming to develop algorithms number of challenges. First, simulating a realistic work set-
that can estimate the level of workload and stress that a ting and inducing stress was challenging. We do think that
knowledge worker is experiencing from sensor data, in order we succeeded in simulating a realistic work setting. Some
to unobtrusively model their mental state. To investigate participants noted afterwards, that although they knew that
whether the stressors affected the participants’ behavior, we the emails were fake, they felt responsible to reply. We also
compared computer interactions and facial expressions in think we were able to manipulate the working conditions
the two stressor conditions with the neutral baseline con- with stressors: the questionnaire data showed that partic-
dition. Under time pressure we see significantly more key ipants’ experience of task load, mental effort and emotion
strokes than in the neutral condition, and under interrup- changed in the stressor conditions. Due to our experimental
tions we see more application changes and left clicks. So design of always starting with the neutral baseline condition,
both stressors create typical behavioral patterns. subjects might have experienced order or fatigue effects. We
Explorative correlation analysis on questionnaire and fa- do, however, think that the relaxation phases helped partic-
cial expression data showed that moderate correlations were ipants to start each condition in a well-rested state. We did
found for mental effort with several facial features. When not find significant effects of our working conditions on per-
working in a condition with higher mental effort, partici- ceived stress. Real-world stress might be complex, involving
pants looked more disgusted and sad and they showed more worries or thing outside work and stress building up over
activation in the facial action units LidTightener, UpperLip- days. Therefore, a limitation of this dataset is that only
Raiser, BrowLowerer and CheekRaiser. So mental effort short term effects of stressors can be investigated. For lon-
might be estimated based upon video information. For more gitudinal research on temporal (stress) patterns over days or
details on our results, see [9]. These analyses show the po- weeks, we are currently recording the presented sensor suite
tential of using the dataset for research on user modeling. in a real-world office.
Context recognition and Information support. The A second challenge was synchronization of all data. Dif-
tasks of writing reports and preparing presentations enabled ferent sensors were recorded via different computers. We
us to use the dataset in the fields of context recognition and synced computer clocks, and most sensors made exact start-
search behavior as well. For this purpose, the raw events ing timestamps upon hitting the record button. Never-
were aggregated in event blocks, which were labeled for their theless, we cannot guarantee second-precise synchronization
task content. Details can be found in [17], where also some among modalities (especially the uEye camera start times
initial analyses of the search behavior in the data are de- may be somewhat unprecise).
scribed. Finally, using different sorts of sensors requires multidisci-
plinary expertise, like knowledge (and software) for process-
ing physiological, image or Kinect data. Our contribution
6. DISCUSSION is to provide a dataset that not only contains raw data, but
To our knowledge, our dataset is the first in which a set of also preprocessed and aggregated data, which makes it easier
unobtrusive sensors from different research fields was used for other researchers to use the data.
to collect data in a realistic office context, while stressors The strength of this dataset is its richness in terms of
were manipulated. modalities and its size in terms of the amount of data per

297
participant. Although limited in size of participants (25), [7] A. Kapoor and R. W. Picard. Multimodal affect
initial experiments have shown that the dataset has suffi- recognition in learning environments. In Proceedings of
cient power to detect significant differences. With the pre- the 13th annual ACM international conference on
sented new dataset, automatic inference of a rich set of con- Multimedia, pages 677–682. ACM, 2005.
text information of a user in the office can be studied. It [8] S. Koldijk. Automatic recognition of context and
can be used to develop context aware systems to support stress to support knowledge workers. In Proceedings of
knowledge workers during their work. Moreover, it provides the 30th European Conference on Cognitive
ample resources for stress and work style related studies. Ergonomics, pages D20–D23. ACM, 2012.
[9] S. Koldijk, M. Sappelli, M. Neerincx, and W. Kraaij.
7. CONCLUSION Unobtrusive monitoring of knowledge workers for
We identified the need of a rich dataset and its desired stress self-regulation. In User Modeling, Adaptation,
characteristics. In this paper we described how we collected and Personalization, pages 335–337. Springer, 2013.
such a new dataset that overcomes drawbacks common in re- [10] S. Koldijk, M. van Staalduinen, M. Neerincx, and
lated work: We used a realistic office setting while stressors W. Kraaij. Real-time task recognition based on
were manipulated systematically. We used a varied set of knowledge workers’ computer activities. In Proceedings
sensors: computer logging, video, Kinect 3D and body sen- of the 30th European Conference on Cognitive
sors. We preprocessed the data and extracted features per Ergonomics, pages 152–159. ACM, 2012.
minute. The resulting dataset SWELL-KW will be shared [11] L. Koppes, E. d. Vroome, G. Mars, B. Janssen,
with the scientific community. We presented a selection of M. Zwieten, and S. v. d. Bossche. Nationale enquete
research questions that could be answered with this dataset. arbeidsomstandigheden 2012: Methodologie en globale
As demonstrated, analyses of the data can yield insights in resultaten, 2012.
the effects of stressors at work, or on the relation between [12] K. Kraan, S. Dhondt, I. Houtman, R. Nelemans, and
subjective ratings and the sensor data. The presented new E. d. Vroome. Handleiding nova-weba: Hernieuwde
affective and behavioral dataset is a valuable contribution versie., 2000.
to research fields like work psychology, user modeling and [13] G. Mark, D. Gudith, and U. Klocke. The cost of
context aware systems. interrupted work: more speed and stress. In
More information on the dataset and its access can be Proceedings of the SIGCHI conference on Human
found at http://persistent-identifier.nl/?identifier= Factors in Computing Systems, pages 107–110. ACM,
urn:nbn:nl:ui:13-kwrv-3e 2008.
[14] F. Mokhayeri, M.-R. Akbarzadeh-T, and
8. ACKNOWLEDGEMENTS S. Toosizadeh. Mental stress detection using
This publication was supported by the Dutch national physiological signals based on soft computing
program COMMIT (project P7 SWELL). techniques. In Biomedical Engineering (ICBME), 2011
18th Iranian Conference of, pages 232–237. IEEE,
9. REFERENCES 2011.
[1] M. M. Bradley and P. J. Lang. Measuring emotion: [15] A. Riera, A. Soria-Frisch, A. Albajes-Eizagirre,
the self-assessment manikin and the semantic P. Cipresso, C. Grau, S. Dunne, G. Ruffini, and
differential. Journal of behavior therapy and S. aStarlab Barcelona. Electro-physiological data
experimental psychiatry, 25(1):49–59, 1994. fusion for stress detection. Studies in health technology
[2] E. Demerouti, A. B. Bakker, S. A. Geurts, and T. W. and informatics, 181:228–32, 2012.
Taris. Daily recovery from work-related effort during [16] J. Ruff. Information overload: Causes, symptoms and
non-work time. Research in Occupational Stress and solutions. Harvard Graduate School of Education,
Well-being, 7:85–123, 2009. pages 1–13, 2002.
[3] E. Demerouti, A. B. Bakker, F. Nachreiner, and W. B. [17] M. Sappelli, S. Verberne, S. Koldijk, and W. Kraaij.
Schaufeli. The job demands-resources model of Collecting a dataset of information behaviour in
burnout. Journal of Applied psychology, 86(3):499, context. In Proceedings of the 4th Workshop on
2001. Context-awareness in Retrieval and Recommendation,
[4] D. F. Dinges, R. L. Rider, J. Dorrian, E. L. 2014.
McGlinchey, N. L. Rogers, Z. Cizman, S. K. [18] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic.
Goldenstein, C. Vogler, S. Venkataraman, and D. N. A multimodal database for affect recognition and
Metaxas. Optical computer recognition of facial implicit tagging. Affective Computing, IEEE
expressions associated with stress induced by Transactions on, 3(1):42–55, 2012.
performance demands. Aviation, space, and [19] L. M. Vizer, L. Zhou, and A. Sears. Automated stress
environmental medicine, 76(Supplement detection using keystroke and linguistic features: An
1):B172–B182, 2005. exploratory study. International Journal of
[5] P. C. Duttweiler. The internal control index: A newly Human-Computer Studies, 67(10):870–886, 2009.
developed measure of locus of control. Educational and [20] D. Zapf. Stress-oriented analysis of computerized office
Psychological Measurement, 44(2):209–221, 1984. work. The European Work and Organizational
[6] S. G. Hart and L. E. Staveland. Development of Psychologist, 3(2):85–100, 1993.
nasa-tlx (task load index): Results of empirical and [21] F. Zijlstra and L. van Doorn. The construction of a
theoretical research. Advances in psychology, scale to measure subjective effort. Technical report,
52:139–183, 1988. Delft University of Technology, 1985.

298