Mdscan - An Explainable Artificial Intelligence Artifact For Menta
Mdscan - An Explainable Artificial Intelligence Artifact For Menta
Mdscan - An Explainable Artificial Intelligence Artifact For Menta
5-2-2023
Anol Bhattacherjee
University of South Florida, [email protected]
Kazim Topuz
The University of Tulsa, [email protected]
Ali Tosyali
Rochester Institute of Technology, [email protected]
Gorden Li
Bosch Center for Artificial Intelligence, [email protected]
Recommended Citation
Tutun, Salih; Bhattacherjee, Anol; Topuz, Kazim; Tosyali, Ali; and Li, Gorden, "MDSCAN: AN EXPLAINABLE
ARTIFICIAL INTELLIGENCE ARTIFACT FOR MENTAL HEALTH SCREENING" (2023). ECIS 2023 Research-in-
Progress Papers. 56.
https://aisel.aisnet.org/ecis2023_rip/56
This material is brought to you by the ECIS 2023 Proceedings at AIS Electronic Library (AISeL). It has been
accepted for inclusion in ECIS 2023 Research-in-Progress Papers by an authorized administrator of AIS Electronic
Library (AISeL). For more information, please contact [email protected].
MDSCAN: AN EXPLAINABLE ARTIFICIAL INTELLIGENCE
ARTIFACT FOR MENTAL HEALTH SCREENING
Research Paper
Salih Tutun, Washington University in St. Louis, St. Louis, Missouri 63130, USA,
[email protected]
Anol Bhattacherjee, University of South Florida, Tampa, Florida 33620, USA,
[email protected]
Kazim Topuz, The University of Tulsa, Tulsa, Oklahoma 74104, USA, kazim-
[email protected]
Ali Tosyali, Rochester Institute of Technology, Rochester, New York 14623, USA,
[email protected]
Gorden Li, Bosch Center for Artificial Intelligence, Sunnyvale, California 94085, USA,
[email protected]
Abstract
This paper presents a novel artifact called MDscan that can help mental health professionals quickly
screen a large number of patients for ten mental disorders. MDscan uses patient responses to the SCL-
90-R clinical questionnaire to create a full-color image, similar to radiological images, which identifies
which disorder or combination of disorders may afflict a patient, the severity of the disorder, and the
underlying logic of this prediction, using an explainable artificial intelligence (XAI) approach. While
prior artificial intelligence (AI) tools have seen limited acceptance in clinical practice because of the
lack of transparency and interpretability in their "black box" models, the XAI approach used in MDscan
is a "white box" model that elaborates which patient feature contributes to the predicted outcome and
to what extent. Using patient data from a mental health clinic, we demonstrate that MDscan outperforms
current (expert-based) clinical practice by an average of 20%.
Keywords: Mental health; Artificial intelligence; Deep learning; Explainable artificial intelligence.
1 Introduction
The world is facing a mental health crisis of unprecedented scale, with an estimated 970 million people,
or nearly one in eight people globally, having lived with mental disorders in the year 2019, a 28%
increase over the previous year (World Health Organization, 2022). This global crisis has worsened
significantly with recent pandemics (e.g., COVID-19) and epidemics (e.g., opioid overdose) (Johnson
et al., 2021). In the United States, one in five adults experience mental disorders today (National Alliance
on Mental Illness, 2022). A mental disorder is defined as a clinically significant disturbance in a person's
cognition, emotional regulation, or behavior that causes significant distress, impairment in functioning,
and self-harm tendencies (World Health Organization, 2022). This includes a wide range of disorders
such as anxiety, depression, obsessive-compulsive disorder, bipolar disorder, schizophrenia,
psychosomatic disorder, autism, paranoia, and post-traumatic stress disorder, each with its unique
configuration of symptoms and requiring a different treatment protocol. According to a study by the
Lancet Commission, the total cost of mental health disorders, in terms of lost productivity, disability,
social welfare, and law and order spending, will exceed $16 trillion worldwide between 2016 and 2030
(The Lancet, 2018).
Despite the growing demand for mental health services, there is an acute shortage of mental health
experts, with some countries reporting as little as one psychiatrist for every 100,000 people (World
Health Organization, 2018). This supply shortage results in long wait times for mental health
appointments, delayed diagnosis and treatment, and continued suffering, even in advanced nations
(Wainberg et al., 2017). Many mental health clinics also lack the tools and resources to screen large
volumes of patients (Kilbourne et al., 2018). Psychiatrists and counselors have therefore called for
innovative technological tools to help augment their capacity to screen and treat patients more efficiently
and effectively (Thieme et al., 2020). In 2019, Health Education England issued a report suggesting
using artificial intelligence (AI) to help augment mental health experts' capacity to screen mental health
patients (Foley and Woollard, 2019).
Modern web technologies make it feasible to administer clinical mental health screening instruments
privately and securely on a large scale and machine learning (ML) techniques can help screen patients
for common mental disorders. However, most ML models are "black box" models that provide little to
no explanation of the rationale behind their prediction. Without explanations, experts cannot trust these
predictions for clinical use, where patients' lives and well-being are often at stake. Explainable artificial
intelligence (XAI), referring to a set of AI/ML tools and techniques that can help human users
understand the reasons for predictions made by AI models (Murdoch et al., 2019), can bring
transparency, explainability, and trust to these black-box models, and make them acceptable for use in
professional settings.
In this paper, we propose a novel XAI artifact called MDscan to help mental health experts screen
patients efficiently and accurately using a "white box" ML approach that explains the reasons for a
diagnosis. MDscan converts tabular patient data from the SCL-90-R mental health questionnaire into
interpretable images that experts can use to screen, classify, and prioritize patients at scale for ten
different mental disorders. Using a computational XAI approach, MDscan generates a full-color visual
representation of ten mental disorders, similar to that of radiological or neurological scans, which were
previously unavailable in mental health practice. We validate our proposed artifact using a field
experiment of 500 patients from a mental health clinic.
2 Related Literature
moderately symptomatic, and severely symptomatic) based on the threshold values of these scores
(Schmitz et al., 2000). However, interpreting SCL-90-R questionnaire data is complex, given the
overlapping nature of symptoms across many mental disorders and the possibility of multiple concurrent
disorders, and is a job usually relegated to trained mental health experts.
2.3 Explainable AI
XAI refers to a set of ML tools and techniques that can help human users understand the reasons behind
the predictions made by AI models (Murdoch et al., 2019). While traditional AI models are typically
“black box” models where even the designers of these models cannot explain why or how their models
arrive at a specific decision, XAI models are considered to be “white box” models that can explain the
key features that contributed to a prediction and the respective weights of those features toward the
prediction (Loyola-Gonzalez, 2019). Such “white box” models give decision-makers the underlying
basis for making decisions and help build transparency and trust in AI models.
The last few years have seen significant progress in XAI models, with the development of many new
packages and methods, such as Local Interpretable Model-agnostic Explanations (LIME), SHapley
Additive exPlanations (SHAP), Shapash, ExplainerDashboard, Dalex, Explainable Boosting Machines
(EBM), ELI5, and others. LIME, proposed by Ribeiro et al. (2016), is one of the earliest XAI techniques
that explains the predictions of a machine learning classifier by learning an interpretable model based
on local observations around the predicted observation. Lundberg and Lee (2017) introduced SHAP to
explain individual predictions globally by assigning each feature a theoretically optimal value for a
particular prediction. We employ these techniques, along with our own ShapRadiation technique and
clinical practice (SCL-90-R), to design an XAI framework for mental health screening.
3 Artifact Design
Our proposed XAI framework uses data from the SCL-90-R questionnaire as inputs and converts these
inputs into explainable two-dimensional images that can be used for mental disorder screening. This
framework consists of three phases: (1) creating a feature map, (2) predictive modeling for image
recognition, and (3) generating an explanatory canvas. Figure 1 depicts an overview of these three
phases.
feature point, which is then set as the following origin of polar coordinates. The ray is rotated
counterclockwise from its new origin until it meets the next feature point, and this process is repeated
iteratively until it reaches back to the starting feature point. The resulting polygon (the inner shape with
red lines in Figure 2) contains all feature points in the least possible canvas space without altering the
relative location of features. Next, we determine the minimum bounding rectangle containing the convex
hull polygon and rotate the minimum bounding rectangle to obtain an orthogonal representation to
minimize the canvas size containing all feature information.
Lastly, we visualize each participant’s SCL-90-R responses using grey-scaled pixels on our minimized
image map by rescaling each response (a 0-4 score) to a pixel value between 0 and 255, using the min-
max normalization function. The higher value is this value, the darker the color of the associated pixel.
Darker pixels are more indicative of a positive prediction of a mental disorder, while lighter pixels reflect
a negative prediction.
after and based on the work of Lloyd Shapley, the 2012 Nobel Prize winner in Economics, who
developed this approach to compute the relative contributions of multiple players in cooperative game
theory. In our problem, each feature represents a player, and the collective contribution of these features
determines the predicted class of a given observation.
MDscan employs the SHAP algorithm to compute Shapley values for each pixel on our feature map
(from Phase 1). The Shapley value of a pixel toward a given class is the difference in probability of
predicting this class using the image containing that pixel versus the image without that pixel (i.e., by
substituting the focal pixel with a random pixel). We estimate ten Shapley values for each pixel to
correspond to our ten classes. Shapley values may be positive or negative, shown in red or blue,
respectively, in Figure 3, with a darker color denoting a value far from zero. A negative Shapley value
indicates a decreased likelihood of categorizing a feature in a specific class, while a positive value
increases the chances of a positive class prediction.
Figure 5. Explanatory canvas with (a) questions and (b) mental disorders.
4 Artifact Evaluation
We tested MDscan using clinical trial data from patients at a mental health clinic in Turkey. Since
September 2018, this clinic has administered the SCL-90-R questionnaire to its clients on a secure web
portal called Psikometrist.com. Counselors at the clinic send clients an encrypted link to the online
questionnaire by e-mail or text for them to complete the questionnaire from the privacy of their homes.
Since its inception, the portal has recorded responses from 15,760 participants. The portal also tracks
how long each participant took to complete each question. To ensure data quality, we removed responses
that were completed in less than two seconds per question or those determined by the clinic staff to
contain inconsistent responses to similar questions. This screening process led to 6,139 complete
observations, which was the initial data source for our artifact evaluation.
The clinic provided us with the actual diagnosis (class labels) for each of the 6,139 patients, which was
used as the “ground truth” for our artifact evaluation. This evaluation was made by clinical psychologists
and psychiatrists based on participants’ SCL-90-R responses, personal interviews, direct observations,
physiological evidence (blood tests, blood pressure, etc.), and medical history. Also available to us was
the General Severity Index (GSI), a measure of patients’ overall mental status, which was essentially
the mean of all 90 patient responses to the SCL-90-R instrument. GSI was used to determine whether a
patient was mentally ill in general and needed treatment.
We selected a random anonymized sample of 500 observations from the 6,139 labeled observations to
evaluate our MDscan algorithm relative to current clinical practice (manual evaluation by mental health
experts based on SCL-90-R responses). We trained three mental health experts (two psychologists and
one psychiatrist) on reading and interpreting the MDscan images. We then asked these experts to
diagnose the 500 patients in our sample using our explainable MDscan images. We compared this
diagnosis against the patients’ original diagnosis reported to us by the clinic. Using two threshold values
for this evaluation (functional versus moderately symptomatic patients, and moderately symptomatic
versus severely symptomatic patients) we computed recall, precision, and F1 scores for our MDscan
approach against the original diagnosis (ground truth). The classification metrics are reported for
functional to moderately symptomatic patients and for moderately to severely symptomatic patients in
Tables 1 and 2, respectively. Clinical practice outperformed MDscan in recall (how well the model
correctly detected true cases) in both samples, but MDscan was superior in precision (how accurate were
the model’s positive predictions). Overall, MDScan consistently outperformed clinical practice on F1
score (the harmonic mean of recall and precision) for all mental disorders by an average of about 20%.
However, the real value of our algorithm was not in outperforming clinical practice but in its ability to
explain the rationale for each diagnosis. Figure 6 illustrates an example of how an MDscan image
displayed features contributing to the prediction of mental disorders. In this figure, the responses to
questions C42, C27, C58, C49, and C12 (in red squares) are darker in color, indicating a high value,
which implies a “positive” prediction of somatization. In contrast, responses to questions C23, C72, and
C86 (in green squares) are lighter in color, indicating a low value and or a “negative” prediction of
anxiety. It is worth noting that using MDscan, mental health experts can diagnose patients with multiple
disorders, which are sometimes missed, given information overload from 90 features and the complex
and overlapping symptomology of many mental disorders.
5 Discussion
This study used an XAI approach to design a novel artifact for screening ten mental disorders, along
with their severity and explanation of the prediction in form of a full-color explanatory image. The
artifact presented here was informed and guided by clinical psychiatric practice (DSM-based SCL-90-
R questionnaire) and latest developments in XAI techniques to help secure user confidence, trust, and
acceptance of this artifact by mental health practitioners for clinical practice. A field test of this artifact
showed that it outperformed standard (manual) clinical practice for each of the ten mental disorders by
an average of 20%. The initial screening provided by MDscan may be complemented with patient
interviews, observations, physiological evidence, and prior medical history to generate a more
comprehensive diagnosis of mental disorders.
6 Conclusion
As the demand for mental health services continues to explode around the globe, there is a growing need
for automated tools to support mental health screening and diagnosis. The MDscan algorithm presented
in this paper attempts to address this critical problem while also demonstrating how to improve user
trust, confidence, and acceptance of AI predictions by explaining their AI predictions. We hope that our
research will inspire other researchers to develop their own XAI approaches for addressing critical
business and social problems in other domains, such as finance and cybersecurity, where visual and
explainable representations of large complex data sets are needed to detect essential and useful patterns.
7 Acknowledgments
The authors thank the Guven Private Health Laboratory, Ankara, Turkey, for providing the labeled
mental disorder data used in this study. This data was sourced with the help of DNB Analytics.
References
American Psychiatric Association (2022). Diagnostic and Statistical Manual of Mental Disorders, Fifth
Edition, Text Revision (DSM-5-TR™). Washington DC.
Foley, T. and Woollard, J. (2019). The digital future of mental healthcare and its workforce. London:
Health Education England.
Graham, R.L. (1972). “An efficient algorithm for determining the convex hull of a finite planar set,”
Information Processing Letters 1, 132-133.
Johnson, M., Albizri, A., and Harfouche, A. (2021). “Responsible artificial intelligence in healthcare:
Predicting and preventing insurance claim denials for economic and social well-being, Information
Systems Frontiers, https://doi.org/10.1007/s10796-021-10137-5.
Kilbourne, A. M., Beck, K., Spaeth-Rublee, B., Ramanuj, P., O’Brien, R. W., Tomoyasu, N., and Pincus,
H. A. (2018). “Measuring and improving the quality of mental health care: A global perspective,”
World Psychiatry 17 (1), 30–38.
Lundberg, S.M. and Lee, S.I. (2017). “A unified approach to interpreting model
predictions,” Proceedings of the 31st Conference on Neural Information Processing Systems, Long
Beach, USA, 4768–4777.