102 Full
102 Full
102 Full
Protected by
The promise of digital mood tracking technologies: are we heading on
the right track?
Gin S Malhi,1,2,3 Amber Hamilton,1,2,3 Grace Morris,1,2,3 Zola Mannie,1,2,3 Pritha Das,1,2,3 Tim Outhred1,2,3
1
Academic Department of Psychiatry, Northern Sydney Local Health District, Sydney, New South Wales, Australia; 2Sydney Medical School
Northern, University of Sydney, Sydney, New South Wales, Australia; 3CADE Clinic, Royal North Shore Hospital, Northern Sydney Local
Health District, Sydney, New South Wales, Australia
Correspondence to Dr Gin S Malhi, Discipline of Psychiatry, Sydney Medical School, University of Sydney, Sydney, NSW 2065, Australia;
gin.malhi@sydney.edu.au
Abstract
The growing understanding that mood disorders are dynamic in nature and fluctuate over variable epochs of time has compelled researchers to
develop innovative methods of monitoring mood. Technological advancement now allows for the detection of minute-to-minute changes while also
capturing a longitudinal perspective of an individual’s illness. Traditionally, assessments of mood have been conducted by means of clinical interviews
and paper surveys. However, these methods are often inaccurate due to recall bias and compliance issues, and are limited in their capacity to collect
and process data over long periods of time. The increased capability, availability and affordability of digital technologies in recent decades has offered
a novel, non-invasive alternative to monitoring mood and emotion in daily life. This paper reviews the emerging literature addressing the use of digital
Clinical review
mood tracking technologies, primarily focusing on the strengths and inherent limitations of using these new methods including electronic self-report,
behavioural data collection and wearable physiological biosensors. This developing field holds great promise in generating novel insights into the
mechanistic processes of mood disorders and improving personalised clinical care. However, further research is needed to validate many of these
novel approaches to ensure that these devices are indeed achieving their purpose of capturing changes in mood.
copyright.
time ‘bodily functions’ can be tracked in real time by individuals them- affect, mood and temperament and the particular epochs in which they
selves. This data can then be simultaneously communicated in real time reside. As depicted in figure 1, an individual’s temperament (akin to
to systems for storage and analysis, allowing for the development of data- personality) is thought to be relatively stable, with modulations occur-
bases sufficiently large and detailed to meaningfully make sense of biom- ring over years and decades, whereas components such as mood and
etric data. Yet, in practice, few of these technologies are used, especially affect vary more frequently and necessarily throughout an individual’s
by clinicians, either for research or day-to-day clinical care. In psychiatric life.1 Affective changes are adaptive responses to external stimuli and as
practice, these technologies have the potential to track the symptoms of such necessarily oscillate rapidly over minutes, seconds or even fractions
mood disorders as devices can log objective measures of mood, arousal, of a second. In contrast, moods are more prolonged emotional states
activity and sleep, and allow users to input subjective experiences. that tend to fluctuate over hours or days1 2 and follow a natural circa-
Additionally, devices have the potential to be part of daily personal care, dian pattern—the nadir usually occurring on awakening.3 The interac-
providing alerts for pharmacological and psychological treatment adher- tion between affect, mood and temperament is a dynamic process as
ence and undertaking physical activity. Promisingly, with further research, a person’s natural temperament may influence their experience of mood
there is the potential for these devices to be embedded within routine states, which may consequently colour their reactions to external stimuli
clinical management, as algorithms for predicting the dynamic nature of (ie, their affective response). However, this interaction may also work in
mood disorder presentations and treatment responses are developed. In the opposite direction, such that experiencing persistent affective insta-
order to progress to this stage, it is necessary to examine the nature of bility is likely to produce changes in mood, which over long periods of time
mood disorders and how existing technologies are being employed to may be inculcated into a person’s temperament. Although this interplay
measure this. This review examines the digital techniques that have been between affect, mood and temperament can be readily conceptualised,
used by researchers to capture the dynamic nature of mood disorders assessing this relationship in practice poses a considerable challenge.
and points to the potential value of these novel technologies in aiding Investigating the interplay between affect, mood and temperament
future clinical management and decision-making. demands an approach that is able to detect granular affective changes
in real time, while also capturing the longitudinal perspective of all these
components. Historically, mood assessments have been conducted by
Methods clinicians, where clinical judgements are based on observations of the
A literature search was performed using databases such as PubMed, patient’s current presentation, as well as subjective reports from patients
PsycINFO and Scopus, with the following search terms: mood tracking, themselves.4 5 Assessed in this manner, the components of affect, mood
depression, mobile phones, biosensors, technology, symptoms, and temperament can be easily conflated. Additionally, reliance on inter-
biomarkers, physiology, smartphones, psychiatry, bipolar disorder. The views to obtain accurate data on mood is inherently flawed as retro-
authors also reviewed literature cited in the bibliographies of related arti- spective self-reported information has been shown to be inaccurate and
cles and review papers. Given the focus of the review on technological highly susceptible to bias.6 More recently, researchers have looked to
advances, it only includes relevant research from the past five years. The reduce the influence of recall bias by increasing the frequency of ques-
paucity of publications in this nascent field has meant that a non-system- tionnaires by employing daily diary studies or administering surveys that
atic search was necessary in order to capture the rapid developments in patients complete at the end of each week.7 However, these types of
this area. pen-and-paper surveys have been associated with disappointingly low
Clinical review
assessments are more frequent, the larger epochs of time they capture a core feature of patients with BPD, using digital technology to assess
are not sufficient to capture the frequent oscillations of affect. Evidently, affect in such short increments revealed that 48% of extreme mood
investigating the interplay between affect, mood and temperament changes occurred within a 15 min time period. Information this detailed
requires a more sophisticated approach to tracking mood than has previ- simply could not have been gathered using traditional surveys or clinical
ously been used. interviews.
Digital technology holds great promise in enhancing our assessment Despite the practical advantages and research benefits that have
of affect, mood and temperament. The increased capability, availability already been seen in adopting digital EMA methods, there are some
and affordability of technology in the form of computers, mobile phones, inherent limitations as to ‘what’ is being assessed. First, although EMA
programmable watches and fitness tools has opened up a variety of attempts to ameliorate the problem of retrospective memory bias when
copyright.
avenues to assess mood.10 For example, the widespread dissemination recalling symptoms, there is still an element of recall involved in elec-
of smartphones has meant that questionnaires are now able to be admin- tronic EMA methods depending on the epoch of time being sampled.13
istered electronically and participant responses can be elicited instan- In order to reduce recall bias further, assessments would need to be far
taneously.11 Furthermore, smartphones are able to collect information more frequent. However, repeatedly administering the same survey and
regarding their user’s call logs, physical activity levels and even their requiring an immediate response could quickly become burdensome and
social media usage.12 Not only can this be collected in real time, improve- intrusive and may influence the accuracy of patient responses.8 Thus, if
ments in storage capabilities mean that data from these devices can be these surveys are to be used for gaining a longitudinal perspective on
easily transmitted, processed and stored with the aim of identifying longi- patients with mood disorders over years of assessment, researchers will
tudinal patterns. Thus, using digital technologies to assess mood is an need to find the balance of collecting precise and timely information in a
attractive prospect for advancing mood disorder research that may help way that is amenable to patients over the long term. The second limita-
elucidate the underlying components of mood disorder psychopathology tion of EMA, more broadly, is that it is still subject to patient self-report.
and their patterns over time. While clinical interviews are based on both verbal reports from patients
This brief review examines the three types of data that may be of use and objective clinical observations, EMA methods rely solely on patient
in mood disorders research: electronic self-report, behavioural data and accounts and self-ratings that are necessarily subjective and vulnerable
physiological measures. to personal bias. Overcoming this issue of subjectivity may require us
to do more than just improving the method of collecting data by using
Electronic self-report digital surveys and rating systems, and might entail the development of
Electronic ecological momentary assessment (EMA) is one such method more creative and novel ways of inferring mood apart from self-reported
adopted from the social psychology sphere that has been valuable to the information.
field of mood disorder research.5 EMA involves the repeated administra-
tion of questionnaires that require an immediate response from partic- The future
ipants.5 While previously being administrated through numerous paper Continuous, real-time monitoring of the physiological, biological and
surveys, technological advances mean that repeated questionnaires can behavioural symptoms of patients with mood disorders may provide
be conveniently delivered on programmed personal digital assistants the objective measures needed to overcome the limitations of EMA and
or mobile phones, thus allowing for a more instantaneous collection of the subjective nature of self-report. Until recently technological restric-
data on participants’ symptoms, affect, mood and behaviour at multiple tions and associated costs have prevented the remote acquisition of
points in time.13 ChronoRecord is one such program that has been widely objective behavioural and physiological data, leaving many unanswered
used in research for the assessment of mania and bipolar depression.14 questions regarding the biological mechanisms underpinning mood
It involves daily self-rating by patients, and rather than completing a disorders. Modern developments such as smartphones with built-in or
questionnaire on symptoms, ChronoRecord requires patients to rate external sensors have enabled the remote capture of biomarker data.
their mood on a visual analogue scale ranging from 0 to 100, which has While mobile apps and programs designed to record electronic EMA and
been validated against some clinical ratings, for example, the Hamilton self-report mood ratings are increasingly common,18 19 there are relatively
Depression Rating Scale and the Young Mania Rating Scale.15 In addition few empirically supported objective tools that validate self-reported data
to rating their mood state, patients are asked to describe features of their and measure potential biomarkers associated with mood disorders.
Facebook, Twitter and emails) and have found that usage patterns and
language choices can be used to infer mood changes.26–28 The real-time Figure 2 Portable and wearable devices. Various electrodes and
and longitudinal data collected from these devices may provide some other sensors can be attached to the body or embedded into everyday
necessary information for investigating the interaction between affect, clothing/accessories. Processing and transmission of data can be
mood and temperament. performed using devices such as smartwatches and smartphones.
Voice and speech analysis is another form of behavioural data that can
be easily recorded by biosensors in smartphone devices. Speech analysis
has been investigated as a passive method of examining mental state and sensors into the fabric to instantaneously record HRV and detect
as psychological states modulate the production of speech.29 Analysis depressive, mixed or hypomanic mood states in bipolar disorder patients.
copyright.
of vocal cues (eg, content, prosody, voice quality and tone of voice) has Roh et al37 have created a low-powered, light-weight headband moni-
been correlated with emotional states.30 31 Several studies investigating toring system with embedded electrodes to record EEG and HRV to clas-
smartphone-based systems proposed that audio, accelerometer and sify mental states in real time. Skin conductance or electrodermal activity
other speech data can be used to classify mood states in bipolar disorder is a biomarker of arousal and has been used to validate self-reported
patients.24 32 33 In particular, Abdullah et al32 found that they could identify arousal levels. Novel systems for measuring electrodermal activity have
mood states in bipolar patients with at least 80% accuracy by using data been developed by Kappeler-Setz et al38 and they have trialled sensor
collected from the patient’s day-to-day phone calls. However, similar to systems in shoes and socks, revealing that recording electrodermal
self-report data this type of data can be susceptible to recall bias. That is, activity through patients’ feet may be a suitable way to record emotional
an individual could be recalling past emotional events when speaking on states and stress levels in bipolar patients (see figure 2).
the phone and therefore vocal cues may not be reflective of their current
emotional state.
Considerations and limitations
While instantaneously measuring and analysing biomarkers has advanced
Wearable physiological data research into the physiological mechanisms underpinning mood state,
To enhance our understanding even further and to overcome some of the there still remain some serious logistical, technological and ethical consid-
limitations of behavioural data collection, researchers in this field have erations within this field (for pros and cons of each type of digital moni-
shifted their focus to utilising the emerging trend of wireless wearable toring device, see table 1). Primarily, smartphone and external biosensors
biosensors. Wearable biosensors allow for the instantaneous collection continuously collect large amounts of data, meaning any analytic
of physiological data such as neural activity, heart rate variability (HRV) approaches must be suitable for ‘big data’ and analysis would require a
and skin conductance to act as objective identifiers of mood states. This large amount of technical and computational expertise. Maintaining the
research follows the current high demand for mobile health and fitness quality of the data limits the execution of research and clinical use but
devices or ‘wearables’ such as fitness watches. also increases the personnel and direct costs. Theinitial development and
The collection of instantaneous physiological data may be an immensely maintenance of biosensor devices and software can be extremely costly.
important step in determining the mechanistic foundation of mood. For Therefore, complicated biosensor systems may not currently be realistic
example, whether these physiological changes are a consequence of in a large-scale, long-term clinical setting. Additionally, due to the volume
mood changes or whether they are the cause of such variability. A recent and type of data collected (ie, sensitive personal, behavioural and physio-
meta-analysis34 has demonstrated that there is a consistent reduction logical information), data security, privacy and confidentiality issues are a
in HRV in patients with psychiatric disorders compared with control major concern which is compounded by the issue of ‘ownership’ over the
patients. HRV can be used as a psycho-physiological biomarker because data. Comprehensive technical consultation and intensive management
it mirrors the variability of heart rate in response to affective and cognitive is essential when developing and using digital monitoring technologies for
states.35 Several groups of researchers have developed various ‘wear- both research and clinical purposes.
able’ monitoring devices to try to capture real-time HRV data in order to Additionally, there are concerns regarding patient adherence to both
improve knowledge of bipolar disorder by identifying potential biomarkers. active and passive digital monitoring programs over the long term, espe-
Valenza et al36 have developed a wearable shirt that integrates electrodes cially when patients experience periods of remission. While this is an
Digital EMA ►► Can easily administer questionnaires and elicit immediate ►► Repeated questionnaires can become burdensome to
responses from participants participants
►► Can use validated rating scales ►► Frequency of EMA can be intrusive
►► Increased compliance compared with pen-and-paper surveys ►► Reliance on self-reported information
Behavioural monitoring
GPS
Location and behaviour tracking ►► Basic technology embedded in normal smartphone devices ►► Location and movement does not immediately translate to
(therefore easily accessible and cheap) social interaction
Phone activity
Internet usage (including social ►► Data are easily recorded ►► Does not necessarily reflect social activity (not face-to-face
media) ►► Non-invasive interactions)
►► Longitudinal monitoring ►► Connectivity issues
Calls logs ►► Data are easily recorded ►► Individuals are typically very unpredictable and variable—hard
►► Non-invasive to determine what may be abnormal for each individual
►► Longitudinal monitoring
Speech analysis ►► Data are easily recorded ►► May be subject to recall bias—voice could be modified due to
►► Non-invasive recalling a previous emotional time
Clinical review
►► Longitudinal monitoring ►► Advanced programs analysis required
Physiological monitoring
Wearables
EEG (eg, headband or hat) ►► Unparalleled access to real-time neural activity ►► Non-compliance, not feasible to wear continuously
►► Advanced programs and analysis required
HRV (eg, shirt or multisensor watch) ►► HRV is a empirically supported psycho-physiological ►► Expensive to create electrodes embedded into shirts
biomarker— immediate HRV feedback is important for research ►► Not feasible to always wear HRV monitoring shirt
and also for monitoring clinical care ►► Not very accurate recording from multisensor watches
►► Advanced programs and analysis required
copyright.
Skin conductance (eg, socks) ►► Easily assess arousal and stress levels ►► Expensive to create and monitor
►► Not feasible to wear all of the time
►► Advanced programs and analysis required
EMA, ecological momentary assessment; GPS, global positioning system; HRV, heart rate variability.
ongoing concern, there has been promising evidence supporting the to effective objective digital mood monitoring. For example, the closer
feasibility of longitudinal use of such digital mood tracking methods. the technological device is to capturing the exact physiological change
For example, digital self-report studies using clinical populations by then the more accurate the formulation will be. Subsequently, current
Bopp et al8 and Tsanas et al39 have reported compliance levels of 75% studies may be underestimating or overestimating certain behaviours due
and >80% respectively, suggesting that digital mood tracking may be to limitations in the technology, variations in the data and further loss of
readily incorporated as part of patient care in the future. While adherence information during analysis. It is important for future research to elucidate
seems to be an issue that can be overcome, another potential ethical these psychometric considerations in developing suitable behavioural
issue that requires further investigation is the concern that repeatedly measures.
asking depressed patients to rate their mood could increase the risk of In order to overcome the limitations of available digital mood tracking
patients focusing on their depression. Further research is required to technologies, further research is necessary to first confirm which valid
establish whether this is a valid ethical concern. objective measures to target and to consequently refine the technology
It must be acknowledged that all of the current biosensor research is to be able to capture that construct as closely as possible.
based on behavioural inferences, such that data collected from digital
biosensors can be mapped on to psychological variables (eg, social
activity, physical activity and mood states).40 It is therefore necessary
to appraise these inferences by examining the validity and reliability of
the measures. Both the construct and external validity of biosensor and
behavioural data must be evaluated to determine whether the objec-
tive measures correlate with the self-report measures and importantly
whether the objective measures capture the intended processes. It is
possible that physiological and behavioural data may only capture the
precursors to emotional state changes or the consequences of change
without tapping into the key construct itself. Each different biosensor or Figure 3 Steps and filters in the translation of physiological
device may be capturing a different aspect of affect, mood or tempera- information to clinical formulation. Pathophysiology of mood occurs
ment, or they may not be capturing the core component of emotional and current technological devices such as smartphones and biosensors
states at all. Currently, at each step along the pathway from pathophys- record certain behavioural and physiological changes. The recorded data
iology to formulation, information is being lost, resulting in our lack of is then analysed, where more of the original information is lost. This
certainty and knowledge surrounding objective measures of mood (see analysis is used for formulation and understanding mood. The majority
figure 3). The proximity of source is currently one of the major barriers of the information is lost by the formulation phase.
16. Morriss RK, Faizal MA, Jones AP, et al. Interventions for helping people recognise
to family members or healthcare professionals. However, many other
early signs of recurrence in bipolar disorder. Cochrane Database Syst Rev
programs are yet to be administered systematically outside of research 2007:CD004854.
studies.43 To date, there has been limited clinical uptake of passive mood 17. Ebner-Priemer UW, Kuo J, Kleindienst N, et al. State affective instability in
monitoring; however, the ‘iSee’ initiative developed by Michigan State borderline personality disorder assessed by ambulatory monitoring. Psychol Med
University44 aims to use students’ behavioural and GPS data to enable 2007;37:961–70.
18. Faurholt-Jepsen M, Ritz C, Frost M, et al. Mood instability in bipolar disorder type
counselling services to engage with target populations more effectively.
I versus type II-continuous daily electronic self-monitoring of illness activity using
Thus, digital self-report programs and passive monitoring have poten- smartphones. J Affect Disord 2015;186:342–9.
tial to greatly enhance the management of mood disorders in the clin- 19. Torous J, Staples P, Shanahan M, et al. Utilizing a Personal Smartphone Custom
ical sphere. However, prior to the widespread adoption of these digital App to Assess the Patient Health Questionnaire-9 (PHQ-9) Depressive Symptoms in
copyright.
tracking technologies in clinical practice, it is imperative that clinicians Patients With Major Depressive Disorder. JMIR Ment Health 2015;2:e8.
20. Place S, Blanch-Hartigan D, Rubin C, et al. Behavioral Indicators on a Mobile Sensing
are consulted as they may be responsible for monitoring, interpreting and
Platform Predict Clinically Validated Psychiatric Symptoms of Mood and Anxiety
acting on the real-time data they receive. Disorders. J Med Internet Res 2017;19:e75.
21. Saeb S, Zhang M, Karr CJ, et al. Mobile Phone Sensor Correlates of Depressive
Symptom Severity in Daily-Life Behavior: An Exploratory Study. J Med Internet Res
Conclusion 2015;17:e175.
Smartphone and other mobile devices, such as wearables, have become 22. Saeb S, Lattie EG, Schueller SM, et al. The relationship between mobile phone
interlaced with our daily lives. It has opened a window to investigate location sensor data and depressive symptom severity. PeerJ 2016;4:e2537.
real-time human behaviour with unprecedented granularity. Access to 23. Ben-Zeev D, Scherer EA, Wang R, et al. Next-generation psychiatric assessment:
such data has already provided researchers with new insights into mood Using smartphone sensors to monitor behavior and mental health. Psychiatr Rehabil J
2015;38:218–26.
tracking and the behavioural underpinnings of mood disorders. With 24. Grünerbl A, Muaremi A, Osmani V, et al. Smartphone-based recognition of
continued multidisciplinary collaboration (eg, with software engineers states and state changes in bipolar disorder patients. IEEE J Biomed Health Inform
and computer scientists) and ever-increasing technological advances, the 2015;19:140–8.
use of smartphone and biosensor tracking will not only provide a deeper 25. Palmius N, Tsanas A, Saunders KEA, et al. Detecting bipolar depression from
understanding of mood states and behaviour, but can also improve clin- geographic location data. IEEE Trans Biomed Eng 2017;64:1761–71.
26. Alvarez-Lozano J, Osmani V, Mayora O, eds. Tell me your apps and I will tell you
ical care for patients with mood disorder. your mood: correlation of apps usage with bipolar disorder state. Proceedings of
Competing interests None declared. the 7th International Conference on PErvasive Technologies Related to Assistive
Environments. ACM, 2014.
Provenance and peer review Not commissioned; externally peer reviewed.
27. De Choudhury M, Gamon M, Counts S, eds. Predicting Depression via Social Media.
doi:10.1136/eb-2017-102757 ICWSM, 2013.
Received 15 June 2017; Revised 31 July 2017; Accepted 9 August 2017 28. Katikalapudi R, Chellappan S, Montgomery F, et al. Associating depressive symptoms
in college students with internet usage using real Internet data. IEEE Technology and
Society Magazine 2012;31:73–80.
29. Hopkins CS, Ratley RJ, Benincasa DS, eds. Evaluation of voice stress analysis
References technology. System Sciences, 2005 HICSS'05 Proceedings of the 38th Annual Hawaii
1. Gray EK, Watson D, Payne R, et al. Emotion, mood, and temperament: Similarities, International Conference on. IEEE, 2005.
differences, and a synthesis. Emotions at work: Theory, research and applications for 30. Chang K-h, Fisher D, Canny J, eds. How’s my mood and stress?: an efficient
management. Chichester: Wiley, 2001:21–43. speech analysis library for unobtrusive monitoring on mobile phones. Proceedings
2. Wilhelm P, Schoebi D. Assessing mood in daily life. Eur J Psychol Assess of the 6th International Conference on Body Area Networks. ICST (Institute for
2007;23:258–67. Computer Sciences, Social-Informatics and Telecommunications Engineering),
3. Boivin DB, Czeisler CA, Dijk DJ, et al. Complex interaction of the sleep-wake cycle 2011.
and circadian phase modulates mood in healthy subjects. Arch Gen Psychiatry 31. Trevino AC, Quatieri TF, Malyska N. Phonologically-based biomarkers for major
1997;54:145–52. depressive disorder. EURASIP J Adv Signal Process 2011;2011:42.
4. Ebner-Priemer UW, Trull TJ. Ecological momentary assessment of mood disorders 32. Abdullah S, Matthews M, Frank E, et al. Automatic detection of social rhythms in
and mood dysregulation. Psychol Assess 2009;21:463–75. bipolar disorder. J Am Med Inform Assoc 2016;23:538–43.
5. Moskowitz DS, Young SN. Ecological momentary assessment: what it is and why 33. Muaremi A, Gravenhorst F, Grünerbl A, eds. Assessing bipolar episodes using speech
it is a method of the future in clinical psychopharmacology. J Psychiatry Neurosci cues derived from phone calls. International Symposium on Pervasive Computing
2006;31:13. Paradigms for Mental Health. Springer, 2014.
Clinical review
copyright.