RESEARCH ARTICLE

Utility of the Child Behavior Checklist as a Screener for Autism

Spectrum Disorder
K. Alexandra Havdahl, Stephen von Tetzchner, Marisela Huerta, Catherine Lord, and Somer L. Bishop

The Child Behavior Checklist (CBCL) has been proposed for screening of autism spectrum disorders (ASD) in clinical
settings. Given the already widespread use of the CBCL, this could have great implications for clinical practice. This
study examined the utility of CBCL profiles in differentiating children with ASD from children with other clinical dis-
orders. Participants were 226 children with ASD and 163 children with attention-deficit/hyperactivity disorder, intel-
lectual disability, language disorders, or emotional disorders, aged 2–13 years. Diagnosis was based on comprehensive
clinical evaluation including well-validated diagnostic instruments for ASD and cognitive testing. Discriminative
validity of CBCL profiles proposed for ASD screening was examined with area under the curve (AUC) scores, sensitiv-
ity, and specificity. The CBCL profiles showed low discriminative accuracy for ASD (AUC 0.59–0.70). Meeting cutoffs
proposed for ASD was associated with general emotional/behavioral problems (EBP; mood problems/aggressive behav-
ior), both in children with and without ASD. Cutoff adjustment depending on EBP-level was associated with
improved discriminative accuracy for school-age children. However, the rate of false positives remained high in chil-
dren with clinical levels of EBP. The results indicate that use of the CBCL profiles for ASD-specific screening would
likely result in a large number of misclassifications. Although taking EBP-level into account was associated with
improved discriminative accuracy for ASD, acceptable specificity could only be achieved for school-age children with
below clinical levels of EBP. Further research should explore the potential of using the EBP adjustment strategy to
improve the screening efficiency of other more ASD-specific instruments. Autism Res 2016, 9: 33–42. V C 2015 Inter-

national Society for Autism Research, Wiley Periodicals, Inc.

Keywords: early detection; diagnosis; emotional/behavioral problems; Child Behavior Checklist (CBCL)

Introduction The Child Behavior Checklist (CBCL) is a well estab-

lished and widely used parent-completed measure of
Diagnosis of ASD is often difficult due to the heteroge- emotional, behavioral, and social problems in children
neity in severity and constellations of ASD symptoms, aged 1.5–5 years and 6–18 years [Achenbach & Rescorla,
variation in symptom presentation with developmental 2000, 2001]. It was developed to assess a range of prob-
level and age, and common co-occurrence of other psy- lem behaviors rather than ASD in particular, and dis-
chiatric conditions. Differential diagnosis is further criminates well between clinic-referred and non-referred
complicated by the fact that social difficulties and children [Achenbach & Rescorla, 2000, 2001]. Recently,
repetitive behaviors are also seen in children with non- the instrument developer proposed that the CBCL is
ASD diagnoses such as attention-deficit/hyperactivity also useful for ASD-specific screening within clinical set-
disorder (ADHD), language disorders, intellectual dis- tings [Achenbach & Rescorla, 2013].
ability (ID), and emotional disorders. Although well- Multiple CBCL scales and profiles have been suggested
validated diagnostic instruments are available to aid in for ASD screening. The CBCL/1.5-5 Withdrawn and Per-
differential diagnosis, in-depth assessment of ASD is vasive Developmental Problems (PDP) scales have been
time intensive and requires clinicians with extensive reported to have high accuracy in distinguishing pre-
training and experience with ASD [Huerta & Lord, schoolers with ASD from preschoolers with other disor-
2012]. This has resulted in an increasing effort to estab- ders (AUC 0.85–0.94) [Muratori et al., 2011; Narzisi et al.,
lish reliable, valid, and cost-efficient instruments that 2013]. The CBCL/6-18 scales Withdrawn/Depressed,
can support clinicians in determining a need for further Social problems, and Thought problems have also been
ASD evaluation. found to differentiate well between school-age children

From the Center for Autism and the Developing Brain, Weill Cornell Medical College, White Plains, New York (K.A.H., M.H., C.L.); Lovisenberg
Diaconal Hospital, Oslo, Norway (K.A.H.); Norwegian Institute of Public Health, Oslo, Norway (K.A.H.); Department of Psychology, University of
Oslo, Oslo, Norway (S.T.); Department of Psychiatry, University of California San Francisco, California (S.L.B.)
Received March 22, 2015; accepted for publication June 13, 2015
Address for correspondence and reprints: Karoline Alexandra Havdahl, Weill Cornell Medical College, Center for Autism and the Developing
Brain, 21 Bloomingdale Road, White Plains, NY, 10605. E-mail: [email protected] or [email protected]
Published online 3 July 2015 in Wiley Online Library (wileyonlinelibrary.com)
DOI: 10.1002/aur.1515
C 2015 International Society for Autism Research, Wiley Periodicals, Inc.
V

INSAR Autism Research 9: 33–42, 2016 33

with ASD and non-ASD disorders [Biederman et al., 2010; study of autism symptoms in children with non-ASD
Duarte, Bordin, de Oliveira, & Bird, 2003; Ooi, Rescorla, diagnoses of ADHD, ID, language disorders, or emo-
Ang, Woo, & Fung, 2011]. However, generalizability of tional disorders. For the current study, the only exclu-
these findings is potentially limited by methodological sionary criteria were incomplete CBCL data (>8 missing
issues, such as exclusion of children with relevant differ- items, n 5 8), and having no DSM-IV-TR disorder
ential diagnoses (e.g., ADHD, language/cognitive impair- (n 5 10). The participants were recruited mainly
ments). Additionally, validity of the results may be through clinic intake/referral and flyers/website com-
limited by sampling children with ASD with a high munication, either in the Division of Developmental
degree of general behavior problems, especially when and Behavioral Pediatrics at Cincinnati Children’s Hos-
comparison is made with a non-ASD group with lower pital Medical Center (CCHMC), or at the University of
levels of behavior problems. Other studies suggest that Michigan Autism and Communications Disorders Cen-
emotional/behavior problems (EBP), such as aggressive ter (UMACC).
behavior and mood problems, contribute substantially to The majority of the parents had some college
elevated scores on other ASD screening tools [Charman (n 5 143) or a higher education level (n 5 166), and
et al., 2007; Hus, Bishop, Gotham, Huerta, & Lord, fewer had H.S. diploma without college (n 5 37) or less
2013]. Hus et al. [2013] suggested that taking non-ASD- (n 5 10, missing n 5 33). No significant difference was
specific behavior problems into account may be needed found between sites in parent education level,
to appropriately interpret scores on ASD screeners. X2 5 2.87, df 5 3, P 5 0.41. The proportion of children
Some studies have found lower accuracy of CBCL pro- of non-white/Caucasian ethnicity was higher at
files in identifying children with ASD in the context of CCHMC (n 5 66, missing n 5 1) compared to at UMACC
children with other clinical problems (AUC 0.67–0.75) (n 5 58, missing n 5 1), X2 5 9.68, df 5 1, P < 0.01.
[Myers, Gross, & McReynolds, 2014; Ooi et al., 2014; The majority of the children without ASD came from
Rescorla, Kim, & Oh, 2014; So et al., 2013]. Comparison CCHMC (88%). Given that these children had previ-
of results across studies is difficult due to variation in ously received non-ASD diagnoses and were not referred
ascertainment methods and limited sample characteri- for ASD concern, they are likely representative of chil-
zation in terms of autism symptom severity, intellectual dren presenting to general developmental disabilities/
ability, and language level. To our knowledge, no previ- psychiatric clinics for assessment. The proportions of
ous studies on this topic have completed in-depth individual non-ASD diagnoses are presented in Table 1.
assessment of ASD for children included in both the The children with ASD (DSM-IV-TR: autistic disorder,
ASD group and the comparison group. Additionally, n 5 156, pervasive developmental disorder-not other-
there is little information about the effect of including wise specified, n 5 65, Asperger syndrome, n 5 5) came
children with previous ASD diagnoses, or about child mainly from UMACC (80%). Nearly half received the
characteristics found to influence other ASD screening ASD diagnosis for the first time through the research
tools, such as intellectual ability, age, and gender [Char- study (preschoolers: 50%, school-aged: 43%).
man et al., 2007; Cholemkery, Mojica, Rohrmann, Gen-
Measures and Procedure
sthaler, & Freitag, 2014; Corsello et al., 2007].
The variability in discriminative accuracy across stud- This research was approved by the Institutional Review
ies clearly warrant further examination of the CBCL’s Board at CCHMC and UMACC. Prior to participation,
validity in distinguishing children with ASD from chil- all caregivers signed an informed consent form.
dren with other clinical disorders. Screening misclassifi- Parents completed the CBCL prior to the diagnostic
cation may lead to inappropriate clinical decisions in evaluation, with a mean time lag of 15 days (SD 5 35).
the assessment process and/or loss of valuable time for The individual CBCL scales are presented in Table 2. All
appropriate interventions [Norris & Lecavalier, 2010]. children underwent a comprehensive clinical evalua-
This study examines the utility of CBCL scales proposed tion, including well-validated diagnostic instruments
for ASD-specific screening to distinguish children with for ASD [i.e., the Autism Diagnostic Interview-Revised,
ASD from children with non-ASD disorders commonly ADI-R; Rutter, LeCouteur, & Lord, 2003 and the Autism
seen in ASD diagnostic clinics. The study also explores Diagnostic Observation Schedule, ADOS; Lord, Rutter,
factors that may help explain the variability in results. DiLavore, & Risi, 1999; Lord et al., 2012], the Vineland
adaptive behavior scales-II [Sparrow, Cicchetti, & Balla,
2005], and cognitive testing: the Differential Ability
Methods
Participants Scales-II [Elliott, 1990; n 5 330] or the Mullen Scales of
Early Learning, [Mullen, 1995; n 5 58]. The assessment
The sample consisted of 407 children aged 2–13 years also included measures relevant for establishing non-
who had been assessed for ASD as part of a research ASD diagnoses, such as the Conners’ Parent Rating

34 Alexandra Havdahl et al./Utility of the CBCL in screening for ASD INSAR

Table 1. Sample Characteristics
CBCL/1.5-5 CBCL/6-18
2
Characteristic ASD (n 5 104) Non-ASD (n 5 57) t/X ASD (n 5 122) Non-ASD (n 5 106) t/X2

Age, years, m (SD) 4.2 (1.1) 4.4 (0.9) 21.4 9.4 (1.8) 9.2 (2.0) 0.8
Gender, male, n (%) 85 (81.7) 43 (75.4) 0.9 91 (74.6) 69 (65.1) 2.4
Nonverbal IQ, m (SD) 76.9 (25.7) 97.6 (20.5) 25.6*** 87.0 (28.3) 90.7 (19.7) 21.2
Verbal IQ, m (SD) 69.5 (32.7) 92.3 (22.1) 25.2*** 81.2 (29.6) 91.3 (22.1) 23.0**
ADOS module, n (%) 14.5** 9.9**
1: Single words or less 54 (51.9) 12 (21.1) 16 (13.1) 2 (1.9)
2: Phrase speech 27 (26.0) 25 (43.9) 9 (7.4) 8 (7.5)
3: Fluent speech 23 (22.1) 20 (35.1) 97 (79.5) 96 (90.6)
ADOS comparison score, m (SD) 7.3 (1.8) 2.3 (2.2) 12.9*** 7.2 (2.2) 2.4 (1.8) 18.3***
High EBP-level, n (%) 34 (32.7) 12 (21.1) 2.4 45 (36.9) 35 (33.0) 0.4
Non-ASD diagnoses, n (%)a 34.3***
ADHD 14 (24.6) 48 (45.3)
Intellectual disability 5 (8.8) 21 (19.8)
Language disorder 33 (57.9) 15 (14.2)
Emotional disorder 5 (8.8) 22 (20.8)

Note. CBCL 5 child behavior checklist, ASD 5 autism spectrum disorder, ADOS 5 autism diagnostic observation schedule, EBP 5 emotional/behavioral
problems, ADHD 5 attention deficit/hyperactivity disorder.
1 preschool ASD case had missing on IQ.
a
Comparison between preschool and school-age non-ASD groups.
*P < 0.05; **P < 0.01; ***P < 0.001.

Scale-Revised [Conners, Sitarenios, Parker, & Epstein, are reported as partial eta squared (g2P ), interpreted as
1998], the Spence Children’s Anxiety Scale [Spence, small: 0.01–0.05, medium: 0.06–0.13, and large: 0.14.
1998], and the Multidimensional Anxiety Scale for Chil- Logistic regression was used to determine whether
dren [March, Parker, Sullivan, Stallings, & Conners, scale combinations resulted in incremental discrimina-
1997]. Following completion of all measures, clinicians tive validity compared with the individual scales. Dis-
met to discuss their impressions and assign a consensus criminative validity was examined using area under the
diagnosis. Although the CBCL was available at time of curve (AUC) scores from nonparametric receiver operat-
diagnosis, this instrument was not used in determining ing curve (ROC) analyses, which is a plot of true posi-
the presence or absence of ASD. tive vs. false positive results. Swets [1988] suggested the
following benchmarks for interpreting AUC scores:
Data Analysis 0.50–0.70 (low accuracy), 0.70–0.90 (moderate accu-
racy), and >0.90 (high accuracy). A sample size calcula-
Analyses were carried out separately for the CBCL/1.5-5 tion, using the StatsToDo website (https://www.
and the CBCL/6-18, using the Statistical Package for statstodo.com/SSizSenSpc_Pgm.php), indicated that 50
Social Sciences (SPSS) version 21. Significance level was cases in each group were needed to detect a difference
set at alpha 5 0.05 (two-tailed). Characteristics of the between chance-level and moderate discrimination
ASD and non-ASD groups were compared using chi (AUC 5 0.50/0.70, a 5 0.05, power 5 0.80). For the pro-
square tests (Fisher’s exact test if cells <5 observations) file demonstrating the highest AUC-score in each age
and t-tests. group, we calculated sensitivity, specificity, and positive
First, we examined whether the CBCL scales suggested likelihood ratio (LR1). Confidence intervals (95%) were
for ASD screening (i.e., Withdrawn, PDP, Withdrawn/ calculated based on the Wilson score method [New-
depressed, Social problems, and Thought problems) combe, 1998]. T scores were used to facilitate compari-
showed diagnostic group differences when controlling son with previous studies.
for other child characteristics. Multivariate Analysis of Stratified analyses were performed to examine whether
Covariance (MANCOVA) was used to examine diagnostic discriminative accuracy was associated with level of EBP,
group differences on (a) composite scales, (b) syndrome ID, and/or previous ASD diagnosis. The CBCL has multi-
scales, and (c) DSM-oriented scales, with gender, nonver- ple scales intended to capture emotional problems (e.g.,
bal IQ, and age as covariates. Raw scores were used in Internalizing, Emotionally reactive, Anxious/depressed,
the MANCOVA, as recommended by Achenbach and Anxiety problems, and Affective problems) and behav-
Rescorla [2000, 2001]. Individual ANCOVAs were only ioral problems (e.g., Externalizing, Attention problems,
analyzed if the MANCOVA was significant. Effect sizes Attention deficit/hyperactivity problems, Oppositional/

INSAR Alexandra Havdahl et al./Utility of the CBCL in screening for ASD 35

Table 2. Mean (SD) Raw Scores on the CBCL/1.525 (N 5 161) and the CBCL/6218 (N 5 228) for Children With ASD and
Non-ASD Disorders
CBCL/1.525 ASD (n 5 104) Non-ASD (n 5 57) F df g2P

Broadband scales* 2.79 3,153 0.05

Total problems* 63.7 (32.7) 50.4 (29.6) 6.36 1,153 0.04
Internalizing** 18.1 (11.2) 13.7 (10.2) 8.28 1,153 0.05
Externalizing 22.6 (11.3) 18.6 (11.4) 3.77 1,153 0.02
Syndrome scales* 2.25 7,149 0.10
Emotionally reactive** 5.3 (4.4) 4.1 (4.1) 7.93 1,149 0.05
Anxious/depressed 3.6 (3.1) 3.3 (3.0) 1.54 1,149 0.01
Somatic complaints 3.4 (2.9) 2.6 (2.6) 3.32 1,149 0.02
Withdrawn** 5.9 (3.3) 3.7 (3.1) 10.35 1,149 0.06
Sleep problems 4.7 (3.7) 4.1 (3.1) 0.86 1,149 0.01
Attention problems 5.2 (2.4) 4.3 (2.9) 1.04 1,149 0.01
Aggressive behavior* 17.5 (9.7) 14.4 (9.5) 4.03 1,149 0.03
DSM-oriented scales* 2.75 5,151 0.08
Affective problems 4.6 (3.1) 3.5 (3.1) 2.60 1,151 0.02
Anxiety problems* 5.1 (4.5) 4.3 (3.6) 4.30 1,151 0.03
PDP** 10.3 (5.4) 7.3 (5.0) 11.89 1,151 0.07
ADHD problems 7.5 (3.0) 6.2 (3.2) 2.78 1,151 0.02
ODD problems 5.9 (3.4) 5.0 (3.6) 3.07 1,151 0.02
CBCL/6218 ASD (n 5 122) Non-ASD (n 5 106) F df g2P
Broadband scales*** 8.95 3,221 0.11
Total problems 62.4 (28.0) 55.2 (29.9) 3.58 1,221 0.02
Internalizing 14.4 (8.2) 12.8 (8.7) 2.54 1,221 0.01
Externalizing 14.6 (10.4) 15.0 (11.4) 0.03 1,221 0.00
Syndrome scales*** 7.75 8,216 0.22
Anxious/depressed 7.0 (5.3) 7.1 (5.2) 0.00 1,216 0.00
Somatic complaints 2.9 (2.9) 2.7 (2.9) 0.78 1,216 0.00
Withdrawn/depressed*** 4.5 (2.8) 3.0 (2.7) 14.98 1,216 0.06
Social problems* 7.8 (4.1) 6.6 (4.1) 5.53 1,216 0.02
Thought problems*** 7.9 (4.9) 5.2 (4.6) 17.16 1,216 0.07
Attention problems 10.9 (4.4) 9.9 (4.7) 2.09 1,216 0.01
Aggressive behavior 11.3 (8.0) 11.0 (8.1) 0.17 1,216 0.00
Rulebreaking behavior 3.3 (3.0) 4.0 (3.8) 2.34 1,216 0.01
DSM-oriented scales 0.48 6,218 0.01
Affective problems 5.2 (3.4) 4.6 (3.6)
Anxiety problems 4.5 (3.3) 4.3 (3.2)
Somatic problems 1.8 (2.1) 1.7 (2.2)
ADHD problems 8.0 (3.6) 8.1 (3.9)
ODD problems 4.4 (2.9) 4.6 (3.0)
Conduct problems 4.9 (5.0) 5.0 (5.4)

Note. CBCL 5 child behavior checklist, ASD 5 autism spectrum disorder, PDP 5 pervasive developmental problems, ADHD 5 attention deficit/hyperac-
tivity, ODD 5 oppositional/defiant, g2P 5 partial eta squared.
1 case excluded from MANCOVA due to missing on IQ.
*P < 0.05; **P < 0.01; ***P < 0.001.

defiant problems). In operationalizing clinically signifi- ular emotional and behavioral scale on this finding.
cant level of EBP, avoiding overlap with core ASD behav- Therefore, EBP-level was operationalized as high when T
iors was a priority. Therefore, scales with item content score (age- and gender-normed) on Aggressive behavior
clearly overlapping with core ASD behaviors were not and/or Affective problems was in the clinical range
considered (e.g., Emotionally reactive, Internalizing). Few (70). For the EBP classification to be useful in children
studies have examined concordance between CBCL scales with problems specific to the emotional or behavioral
and co-occurring emotional/behavioral disorders in chil- domain, high EBP was defined as scoring in the clinical
dren with ASD. An exception is a recent study of school- range on either of the scales (results were very similar
aged children with ASD, finding the highest discrimina- when using only one of the scales).
tive validity for the Affective problems and Aggressive All results should be interpreted in light of their con-
behavior scales (AUC 5 0.90) [Gjevik, Sandstad, Andreas- fidence intervals. Charman et al. [2007] found a differ-
sen, Myhre, & Sponheim, 2015]. To avoid the multiple ence in specificity of 0.41 and 0.93 for another ASD
comparisons problem, we based the choice of the partic- screener between subgroups with high and low EBP. A

36 Alexandra Havdahl et al./Utility of the CBCL in screening for ASD INSAR

Table 3. Mean (SD) T Scores and Area Under the Curve (AUC) Scores for ASD Screening Scales
Preschool CBCL/1.5-5 ASD (n 5 104) Non-ASD (n 5 57) AUC 95% CI SE
Withdrawn 68.9 (10.9) 62.0 (10.3) 0.69*** 0.61–0.78 0.04
PDP 71.0 (11.3) 64.0 (11.3) 0.68*** 0.59–0.76 0.05
School-age CBCL/6-18 ASD (n 5 122) Non-ASD (n 5 106) AUC 95% CI SE
Withdrawn/depressed 64.7 (8.5) 59.8 (8.4) 0.67*** 0.60–0.74 0.04
Thought problems 68.7 (9.5) 62.9 (10.0) 0.67*** 0.60–0.74 0.04
Social problems 66.6 (9.4) 63.9 (9.4) 0.59* 0.51–0.66 0.04
WTP scale 133.5 (13.9) 122.7 (15.8) 0.70*** 0.63–0.77 0.04
Note. ASD 5 autism spectrum disorder, CBCL 5 Child behavior checklist, PDP 5 pervasive developmental problems, WTP 5 Withdrawn-Thought Prob-
lems (aggregated T scores), CI 5 confidence interval, SE 5 standard error.
*P < 0.05; **P < 0.01; ***P < 0.001.

sample size calculation indicated that 13 cases in each Overall Discriminative Validity
group were needed to have 80% power to detect a dif-
As shown in Table 3, overall discriminative validity of
ference of this size (a 5 0.05; StatsToDo).
the two CBCL/1.5-5 scales proposed for ASD screening
was in the low range (AUC 0.68–0.69). Logistic regres-
Results sion showed no incremental discriminative value of
Sample Characteristics combining the scales. Only Withdrawn made a signifi-
cant unique contribution to discrimination (B 5 0.22,
As shown in Table 1, there were large differences in
P 5 0.01), while the nonoverlapping items from PDP
ADOS scores between the ASD and non-ASD groups.
did not contribute significantly (B 5 0.00, P 5 0.99),
The ASD group also showed lower intellectual ability,
v2(2) 5 15.02, P < 0.01. Due to similar findings, further
with significant differences in verbal IQ in both age
results are only presented for Withdrawn.
samples, and in nonverbal IQ in the preschool sample.
The CBCL/6-18 scales suggested for ASD screening also
No significant differences were found for age or gender
resulted in AUC-scores in the low range (AUC 5 0.59–
proportions. Among children with non-ASD disorders,
0.67). Logistic regression showed that combining the
the proportion with language disorders was higher in
scales had incremental discriminative value compared to
the preschoolers, whereas the proportion with ADHD
the individual scales. Withdrawn/depressed and Thought
and emotional disorders was higher in the school-age
problems made statistically significant unique contribu-
children. The prevalence of high EBP was 33% in the tions to discrimination (B 5 0.06, P < 0.01 and B 5 0.05,
total sample, with no significant differences between P < 0.01, respectively), whereas Social problems did not
the ASD and non-ASD groups. The two scales compris- contribute significantly (B 5 20.02, P 5 0.32), v2(3)
ing EBP-level did not significantly correlate with age, 5 29.04, P < 0.01. The aggregated scale of T scores from
nonverbal IQ, or verbal IQ (Pearson’s r ranged from Withdrawn/depressed and Thought problems, hereafter
20.09 to 0.09, P  0.16). referred to as Withdrawn-Thought Problems (WTP),
Group Differences on the CBCL yielded an AUC-score of 0.70.
Given the site differences between the ASD and non-
Table 2 presents mean raw CBCL scores and MANCOVA ASD groups, we examined the possible covariate effect
results for the ASD and non-ASD groups (mean T scores of site (UMACC vs. CCHMC) using ROC regression in
are provided as supplementary information). Control- Stata version 13. Site did not show a significant covari-
ling for gender, age, and nonverbal IQ, preschoolers ate effect on either the preschool Withdrawn scale
with ASD scored significantly higher than preschoolers (P 5 0.87) or the school-age WTP scale (P 5 0.85).
with non-ASD disorders on Withdrawn and PDP
Sensitivity, Specificity, and Likelihood Ratio
(medium effect sizes, ES). The ASD group also scored
significantly higher on Total problems, Internalizing, Sensitivity, specificity, and LR1 of the Withdrawn and
Emotionally reactive, Aggressive behavior, and Anxiety WTP scales was examined at two previously suggested T
problems (small ES). In the school-age sample, the ASD score cutoffs of 65 and 62 [Muratori et al., 2011;
group scored significantly higher than the non-ASD Narzisi et al., 2013], using the aggregated mean scale
group only on the scales suggested for ASD screening cutoff when combining scales (130 and 124 for
(i.e., Withdrawn/depressed, Social problems, and WTP) [Biederman et al., 2010]. At the higher cutoff con-
Thought problems, small-to-medium ES), controlling sistent with the CBCL “borderline clinical” cut-point,
for gender, age, and nonverbal IQ. sensitivity and specificity was 63% (95% CI 5 53–73)

INSAR Alexandra Havdahl et al./Utility of the CBCL in screening for ASD 37

Table 4. Area Under the Curve (AUC) Scores, Sensitivity, Specificity and Positive Likelihood Ratio (95% Confidence Inter-
vals) of the Withdrawn and WTP Scales in the Total Sample and in Subgroups
CBCL/1.5-5 Withdrawn (T score 62) AUC Sensitivity, % Specificity,% Likelihood ratio1

Stratification (n ASD/n Non-ASD)

Total sample (104/57) 0.69 (0.61–0.78) 74 (64–82) 53 (39–66) 1.6 (1.2–2.1)
High EBP-level (34/12) 0.62 (0.43–0.81) 88 (72–96) 8 (0–40) 1.0 (0.8–1.2)
Low EBP-level (70/45) 0.70 (0.61–0.80) 67 (55–78) 64 (49–78) 1.9 (1.2–2.9)
Previous ASD diagnosis (52) 0.74 (0.65–0.84) 81 (67–90) 1.7 (1.3–2.3)
No previous ASD diagnosis (52) 0.64 (0.54–0.75) 67 (53–79) 1.4 (1.0–2.0)
CBCL/6-18 WTP (Aggregated T score 124) AUC Sensitivity,% Specificity,% Likelihood ratio1
Stratification (n ASD/n Non-ASD)
Total sample (122/106) 0.70 (0.63–0.77) 78 (69–85) 55 (45–64) 1.7 (1.4–2.2)
High EBP-level (45/35) 0.62 (0.49–0.74) 96 (84–99) 6 (1–13) 1.0 (0.9–1.1)
Low EBP-level (77/71) 0.79 (0.72–0.86) 68 (56–78) 79 (67–87) 3.2 (2.0–5.1)
No ID (93/85) 0.73 (0.66–0.81) 80 (70–87) 56 (45–67) 1.8 (1.4–2.4)
ID (29/21) 0.59 (0.42–0.76) 72 (53–87) 48 (26–70) 1.4 (0.9–2.2)
Previous ASD diagnosis (70) 0.70 (0.62–0.77) 80 (68–88) 1.8 (1.4–2.2)
No previous ASD diagnosis (52) 0.71 (0.63–0.79) 75 (61–86) 1.7 (1.3–2.2)

Note. CBCL 5 Child behavior checklist, ASD 5 autism spectrum disorder, WTP 5 Withdrawn-Thought Problems, EBP 5 emotional/behavioral problems,
ID 5 intellectual disability.

and 65% (95% CI 5 51–77) for Withdrawn, and 58% erate range for children with low EBP (AUC 5 0.70–
(95% CI 5 50–68) and 68% (95% CI 5 58–76) for WTP, 0.79) and in the low range for children with high EBP
respectively. LR1 was 1.8 for both Withdrawn (95% (AUC 5 0.62).
CI 5 1.2–2.7) and WTP (95% CI 5 1.3–2.5). With regard to the CBCL/6-18 WTP, scores at or
The lower cutoff resulted in moderate sensitivity above 124 were associated with a 3.2 increase in likeli-
(74% for Withdrawn, 78% for WTP) and low specificity hood of ASD among children with low EBP, in contrast
(53% for Withdrawn, 55% for WTP). Change in proba- to no increase among children with high EBP (1.0).
bility of ASD diagnosis given scores above the lower Optimal cutoffs (maximized specificity with sensitivity
cutoff was small both for Withdrawn (1.6) and WTP 80%) were widely differing in children with high
(1.7). The cutoff required to identify at least 80% of compared to low EBP-level. In the low EBP subgroup, a
children with ASD resulted in specificity of 39% for cutoff of 117 correctly classified 82% (95% CI 5 71–89)
Withdrawn (95% CI 5 26–51, cutoff 58) and 53% for of children with ASD and 62% (95% CI 5 50–73) of chil-
WTP (95% CI 5 43–63, cutoff 123). dren with non-ASD disorders. For children with high
EBP, compared to cutoff 124, a cutoff of 134 resulted in
Factors Associated With Discriminative Validity
improved specificity from 6% (95% CI 5 1–13) to 40%
Table 4 presents the results of the subgroup analyses for the (95% CI 5 24–58) while maintaining sensitivity at 81%
more sensitive lower cutoff by level of EBP, ID, and previ- (95% CI 5 67–91) (see Fig. 1).
ously/first diagnosed ASD. Subgroup analysis by gender was Although a similar pattern was found for the CBCL/
attempted, but was not possible due to confounding of gen- 1.5-5 Withdrawn, CIs were wider, especially in the
der and high EBP within children with ASD, with signifi- small high EBP subgroup (n 5 46). In the larger low EBP
cantly higher proportion of EBP in girls compared to boys in subgroup (n 5 115), discriminative accuracy was some-
preschoolers (53% vs. 29%), v2(1, N 5 104) 5 4.20, P 5 0.04, what lower than for the school-age low EBP subgroup
and school-age children (55% vs. 31%), v2(1, N 5 122) 5 (AUC 0.70 vs. 0.79). The cutoff required to identify at
5.75, P 5 0.02). There was no significant difference in the least 80% of preschoolers with ASD in the low EBP sub-
proportions of high EBP between girls and boys with non- group, resulted in only 33% specificity (cutoff 54, sensi-
ASD disorders in preschoolers (14% vs. 23%), Fisher’s exact tivity: 87%). Thus, it was not possible to achieve
P 5 0.71, or in school-aged children (32% vs. 33%), v2(1, acceptable discriminative accuracy by using adjusted
N 5 106) 5 0.01, P 5 0.93). cutoffs.
Level of EBP Intellectual Disability

Discriminative utility of the Withdrawn and WTP Due to few children with ID in the preschool non-ASD
showed substantial variability depending on EBP-level. group (n 5 5), this analysis was only performed for the
For both scales, discriminative validity was in the mod- school-age sample. Although discriminative accuracy of

38 Alexandra Havdahl et al./Utility of the CBCL in screening for ASD INSAR

 that the CBCL has been proposed for screening rather
than diagnosis, sensitivity may be considered the high-
est priority. The cutoff required to identify at least 80%
of children with ASD in this study was lower than
found in previous studies. Compared to reported sensi-
tivity of 78–90% [Biederman et al., 2010; Myers et al.,
2014; Narzisi et al., 2013], sensitivity in this study was
58–63% at the threshold consistent with the CBCL
“borderline clinical” problems cutoff (65 for individ-
ual narrow-band scales; average scale score for scale
combinations). Limited sample characterization in pre-
vious studies makes comparison difficult, which is prob-
lematic given that sample characteristics influence our
Figure 1. Sensitivity and specificity (%) of the WTP scale in ability to predict screening efficiency in the intended
children with high EBP (n 5 80) at cutoff 124 and 134. Abbrevi- population. Biederman et al. [2010] reported higher
ation: EBP 5 emotional/behavioral problems. sensitivity in their ASD sample characterized by a high
level of general behavior problems, consistent with the
the WTP was in the moderate range for children with- subgroup showing the highest sensitivity in this study.
out ID (AUC 5 0.73) and in the low range for children In some studies, lack of representation of children with
with ID (AUC 5 0.59), the CIs were highly overlapping. milder ASD presentations (i.e., DSM-IV/ICD diagnoses
other than autistic disorder) is likely to have contrib-
Previously Diagnosed ASD uted to higher sensitivity estimates [Myers et al., 2014;
Limiting the preschool ASD group to previously diag- Ooi et al., 2011].
Utility of the CBCL to identify children in need of
nosed vs. children diagnosed for the first time, discrimi-
further ASD assessment requires specificity within
native accuracy of the Withdrawn scale was in the
acceptable limits with regard to resources needed to
moderate (AUC 5 0.74) and low range (AUC 5 0.64),
resolve false positive cases and potential loss of time for
respectively. Sensitivity of the lower cutoff was within
appropriate interventions. At the threshold necessary to
acceptable limits (80%) only for preschoolers with pre-
identify at least 80% of children with ASD, specificity
vious ASD diagnoses. However, the CIs of the estimates
was low (39–53%). This is consistent with low-to-
overlapped.
moderate specificity found for CBCL profiles in two
WTP differentiated school-age children with and
other studies that included a range of non-ASD disor-
without ASD similarly when the ASD group was limited
ders [Myers et al., 2014; So et al., 2013]. The results
to children previously diagnosed (AUC 5 0.70) as to
indicate that the CBCL scales would likely result in a
children first diagnosed (AUC 5 0.71).
large number of false positives if used to screen for ASD
in clinical settings. False positive screening could lead
Discussion to a narrowing of assessment focus, possibly at the
expense of more appropriate alternatives. Resolving
Children with ASD scored significantly higher than false positive cases can cost valuable time and resources
children with non-ASD disorders on CBCL scales pro- and/or delay delivery of appropriate interventions.
posed for ASD screening (i.e., Withdrawn, PDP, With- Additionally, unwarranted referrals to ASD specialty
drawn/depressed, Social problems, and Thought clinics could give rise to needless emotional distress
problems), when controlling for other child characteris- and economic expenses for families [Sikora, Hall, Hart-
tics. The CBCL/1.5-5 scales Withdrawn and PDP showed ley, Gerrard-Morris, & Cagle, 2008].
similar differentiation, whereas a combination of the In line with previous findings for other ASD screening
CBCL/6-18 scales Withdrawn/depressed and Thought tools [Charman et al., 2007], specificity was especially
problems differentiated best. However, the scales low in children with high EBP, with 74–92% of children
showed low discriminative validity when used to distin- with non-ASD disorders misclassified when using pro-
guish between individual children with ASD and non- posed cutoffs. Although statistical control is not avail-
ASD disorders (AUC 0.59–0.70). Scores above previously able in clinical practice, clinicians may nevertheless
suggested cutoffs were associated with only a small need to take into account the level of EBP when inter-
increase in probability of ASD diagnosis (all 1.8). preting ASD screening results. In this study, the age and
There is an inherent tradeoff between maximizing gender normed CBCL scales Affective problems and
sensitivity and minimizing false positives, and priority Aggressive behavior were used to define an easily appli-
depends on the purpose of the instrument. Considering cable indicator of high EBP (either scale 70). The

INSAR Alexandra Havdahl et al./Utility of the CBCL in screening for ASD 39

optimal cutoff maximizing specificity with high sensi- higher in the ASD or in the non-ASD comparison
tivity (80%) differed widely between the subgroups group, overall estimates of discriminative accuracy
stratified by EBP-level. For the WTP, use of EBP-level could be overestimated or underestimated, respectively.
specific cutoffs resulted in greatly improved specificity Although there was no significant difference on the
in children with high EBP, while maintaining sensitiv- EBP-level classifier between the diagnostic groups in
ity above 80% in both EBP subgroups and with 62% this study, the preschoolers with ASD had somewhat
specificity in children with low EBP. Although this higher scores on several emotional/behavioral scales
strategy led to substantially improved discriminative including Aggressive behavior compared to preschoolers
accuracy, the rate of false positives was still high in without ASD. Thus, given that higher EBP would be
children with high EBP. expected to be associated with higher likelihood of
Although EBP-level also seemed to moderate the dis- meeting cutoffs on the Withdrawn scale, the poor over-
criminative accuracy of the preschool Withdrawn scale all discriminative accuracy in the preschool age group
(i.e., differing likelihood ratios by EBP-level), it was not is perhaps even more concerning.
possible to achieve similar overall improvement of dis- This study adds to the literature on this topic in sev-
criminative accuracy with the use of EBP-level-specific eral ways, including (a) use of a well-characterized sam-
cutoffs. Unlike the school-age WTP scale, the preschool ple of children with ASD and children with previous
Withdrawn scale showed poor discriminative accuracy diagnoses of non-ASD disorders who all completed a
among children without clinically significant EBP. comprehensive diagnostic evaluation of ASD; (b) explor-
There may be several explanations for the variability in ing factors that may help explain the variability in
discriminative accuracy between the preschoolers and results across studies; and (c) presenting a strategy for
school-aged children with low EBP. First, different taking EBP-level into account to improve the discrimina-
scales were used for the two age groups with only the tive accuracy for ASD. The results must also be inter-
school-age WTP including items related to the repeti- preted in light of some methodological limitations. The
tive behavior symptom domain (e.g., “Repeats certain sample consisted mainly of children with relatively high
acts over and over,” “Can’t get his/her mind off certain intellectual ability, and our findings may not generalize
thoughts”). The lack of representation of this core ASD to more cognitively impaired children. However, given
symptom domain could help explain the poor discrim- that the CBCL has not been normed for children with
inative accuracy even in the low EBP subgroup. ID, the sample may be especially relevant to the popula-
Another contributing factor could be the relatively tion for which it is intended. In common with Bieder-
higher proportion of children with language disorders man et al. [2010], the sample included children with
in the preschool compared to the school-aged non- previous diagnoses, and child behavior rating may differ
ASD group. Withdrawn has been found to be the most depending on whether parents are aware of the presence
commonly elevated narrow-band CBCL scale in pre- of a diagnosis. However, if previous diagnosis leads to
schoolers with language disorders [Maggio et al., more parent awareness of behaviors associated with the
2014]. Third, given that the choice of the particular particular diagnosis, this should have contributed to
EBP scales was based on research with school-aged chil- higher discriminative accuracy for ASD rather than
dren [Gjevik et al., 2015], alternative EBP classifications lower, further supporting our finding of low overall
could potentially be more useful for defining adjusted accuracy. Notably, due to small subgroups reflected in
cutoffs on ASD screening scales in preschoolers. wide confidence intervals, power was limited and repli-
Finally, because problem behaviors and symptoms may cation with larger samples is needed to yield precise esti-
be less differentiated in very young children, it is possi- mates. Another limitation is that the screening scales
ble that adjusting for EBP-level has less utility for and the EBP classification were derived from the same
improving discriminative accuracy of ASD screeners in instrument. Finally, given that there is little knowledge
preschoolers compared to older children. Future stud- about which CBCL scales are most accurate in capturing
ies could examine this with the use of the same ASD EBP in children with and without ASD and at different
screener across age groups. age levels, future studies should examine which scales
The results of this study demonstrate the importance and cutoffs are most useful for determining EBP-levels.
of taking moderating factors such as EBP-level into Due to the widespread use of the CBCL in clinical set-
account when evaluating the discriminative validity of tings worldwide, reports of its utility for ASD-specific
screening tools for ASD. Given that high EBP was asso- screening could have substantial implications for prac-
ciated with increased likelihood of meeting cutoffs on tice. Although the CBCL is useful in providing informa-
the scales proposed for ASD screening, estimates of dis- tion about a range of behavioral functions and for
criminative accuracy could vary according to the distri- identifying children with behavior problems (first level
bution of EBP-level in a particular sample [see Janes & screening), the results of this study suggest that the
Pepe, 2008]. Depending on whether the rate of EBP is CBCL scales are not useful for ASD-specific screening.

40 Alexandra Havdahl et al./Utility of the CBCL in screening for ASD INSAR

Although adjustment for EBP-level improved specificity, disorders. The British Journal of Psychiatry, 191(6), 554–
it was not possible to achieve acceptable levels of sensi- 559.
tivity and specificity due to moderate discriminative Cholemkery, H., Mojica, L., Rohrmann, S., Gensthaler, A., &
Freitag, C.M. (2014). Can autism spectrum disorders and
validity even within subgroups of children with low
social anxiety disorders be differentiated by the Social
EBP, without ID, and with previous ASD diagnoses.
Responsiveness Scale in children and adolescents?. Journal
However, the strategy of using the CBCL to define EBP- of Autism and Developmental Disorders, 44(5), 1168–1182.
level and applying EBP-level specific cutoffs could Conners, C.K., Sitarenios, G., Parker, J.D.A., & Epstein, J.N.
potentially improve the screening efficiency of other (1998). The revised Conners’ Parent Rating Scale (CPRS-R):
tools that are more ASD-specific, such as the Social Factor structure, reliability, and criterion validity. Journal of
Responsiveness Scale (Constantino & Gruber, 2005) or Abnormal Child Psychology, 26(4), 257–268.
the Social Communication Questionnaire (Rutter, Bai- Constantino, J.N., & Gruber, C. (2005). The social responsive-
ley, & Lord, 2003). It may also be possible to improve ness scale. Los Angeles, CA: Western Psychological Services.
the discriminative validity of diagnostic instruments for Corsello, C., Hus, V., Pickles, A., Risi, S., Cook, E.H., Leventhal,
B.L., & Lord, C. (2007). Between a ROC and a hard place:
ASD, such as the ADI-R and the ADOS, by taking EBP-
Decision making and making decisions about using the SCQ.
level into account. Future research is needed to exam-
Journal of Child Psychology and Psychiatry, 48(9), 932–940.
ine this.
Duarte, C.S., Bordin, I.A., de Oliveira, A., & Bird, H. (2003).
The CBCL and the identification of children with autism
Acknowledgments
and related conditions in Brazil: Pilot findings. Journal of
C.L. and S.L.B. receive royalties for the sale of diagnos- Autism and Developmental Disorders, 33(6), 703–707.
tic instruments they have co-authored (ADOS, ADOS-2, Elliott, C. (2007). Differential ability scales (2nd ed.). San Anto-
ADI-R). They both donate all royalties related to any nio, TX: Harcourt Assessment.
Gjevik, E., Sandstad, B., Andreassen, O.A., Myhre, A.M., &
research or clinical activities in which they are involved
Sponheim, E. (2015). Exploring the agreement between
to charity. K.A.H, M.H., and S.T. report no conflicts of
questionnaire information and DSM-IV diagnoses of comor-
interest. This work was supported by the South-Eastern bid psychopathology in children with autism spectrum dis-
Norway Regional Health Authority (2012101 to K.A.H) orders. Autism, 19(4), 433–442.
and the National Institutes of Health (R01HD065277 to Huerta, M., & Lord, C. (2012). Diagnostic evaluation of autism
S.L.B; RC1MH089721 and R01MH081873-01A1 to C.L). spectrum disorders. Pediatric Clinics of North America,
The authors are grateful to all of the participating fami- 59(1), 103–111.
lies and to the clinical research staff. We thank Anne- Hus, V., Bishop, S., Gotham, K., Huerta, M., & Lord, C. (2013).
Siri Øyen, Camilla Stoltenberg, Synnve Schjølberg, Factors influencing scores on the Social Responsiveness Scale.
Shanping Qiu, Erin Molloy, and Vanessa Hus Bal for Journal of Child Psychology and Psychiatry, 54(2), 216–224.
Janes, H., & Pepe, M.S. (2008). Adjusting for covariates in stud-
technical assistance with the data and preparation of
ies of diagnostic, screening, or prognostic markers: An old
the manuscript.
concept in a new setting. American Journal of Epidemiol-
ogy, 168(1), 89–97.
References Lord, C., Rutter, M., DiLavore, P.C., Risi, S., Gotham, K., &
Bishop, S.L. (2012). Autism Diagnostic Observation Sched-
Achenbach, T.M., & Rescorla, L.A. (2000). Manual for the ule–2nd edition (ADOS-2). Torrance, CA: Western Psycho-
ASEBA preschool forms & profiles. Burlington, VT: Univer- logical Services.
sity of Vermont, Research Center for Children, Youth, & Lord, C., Rutter, M., DiLavore, P.C., & Risi, S. (1999). Autism
Families. Diagnostic Observation Schedule (ADOS). Los Angeles, CA:
Achenbach, T.M., & Rescorla, L.A. (2001). Manual for the Western Psychological Services.
ASEBA school-age forms & profiles. Burlington, VT: Univer- Maggio, V., Gran ~ ana, N.E., Richaudeau, A., Torres, S.,
sity of Vermont, Research Center for Children, Youth, & Giannotti, A., & Suburo, A.M. (2014). Behavior problems in
Families. children with specific language impairment. Journal of
Achenbach, T.M., & Rescorla, L.A. (2013). Achenbach system Child Neurology, 29(2), 194–202.
of empirically based assessment. In Volkmar, F. (Ed.), Ency- March, J.S., Parker, J.D.A, Sullivan, K., Stallings, P., & Conners,
clopedia of autism spectrum disorders (pp. 31–39). New C.K. (1997). The Multidimensional Anxiety Scale for Chil-
York, NY: Springer. dren (MASC): Factor structure, reliability, and validity. Jour-
Biederman, J., Petty, C.R., Fried, R., Wozniak, J., Micco, J.A., nal of the American Academy of Child & Adolescent
Henin, A., Faraone, S.V. (2010). Child Behavior Checklist Psychiatry, 36(4), 554–565.
clinical scales discriminate referred youth with autism spec- Muratori, F., Narzisi, A., Tancredi, R., Cosenza, A., Calugi, S.,
trum disorder: A preliminary study. Journal of Developmen- Saviozzi, I., . . . Calderoni, S. (2011). The CBCL 1.5–5 and
tal & Behavioral Pediatrics, 31(6), 485–490. the identification of preschoolers with autism in Italy. Epi-
Charman, T., Baird, G., Simonoff, E., Loucas, T., Chandler, S., demiology and Psychiatric Sciences, 20(04), 329–338.
Meldrum, D., & Pickles, A. (2007). Efficacy of three screen- Mullen, E.M. (1995). Mullen scales of early learning. Circle
ing instruments in the identification of autistic-spectrum Pines, MN: American Guidance Service.

INSAR Alexandra Havdahl et al./Utility of the CBCL in screening for ASD 41

Myers, C.L., Gross, A.D., & McReynolds, B.M. (2014). Broad- Rutter, M., LeCouteur, A., & Lord, C. (2003). Autism Diag-
band behavior rating scales as screeners for autism?. Journal nostic Interview–Revised. Los Angeles, CA: Western Psycho-
of Autism and Developmental Disorders, 44(6), 1403–1413. logical Services.
Narzisi, A., Calderoni, S., Maestro, S., Calugi, S., Mottes, E., & Sikora, D.M., Hall, T.A., Hartley, S.L., Gerrard-Morris, A.E., &
Muratori, F. (2013). Child Behavior Check List 11=2–5 as a tool to Cagle, S. (2008). Does parent report of behavior differ across
identify toddlers with autism spectrum disorders: A case-control ADOS-G classifications: Analysis of scores from the CBCL
study. Research in Developmental Disabilities, 34(4), 1179–1189. and GARS. Journal of Autism and Developmental Disorders,
Newcombe, R.G. (1998). Interval estimation for the difference 38(3), 440–448.
between independent proportions: Comparison of eleven So, P., Greaves-Lord, K., Van der Ende, J., Verhulst, F.C.,
methods. Statistics in Medicine, 17(8), 873–890. Rescorla, L., & de Nijs, P.F. (2013). Using the Child Behav-
Norris, M., & Lecavalier, L. (2010). Screening accuracy of level ior Checklist and the Teacher’s Report Form for identifica-
2 autism spectrum disorder rating scales: A review of tion of children with autism spectrum disorders. Autism,
selected instruments. Autism, 14(4), 263–284. 17(5), 595–607.
Ooi, Y.P., Rescorla, L., Ang, R.P., Woo, B., & Fung, D.S. (2011). Sparrow, S., Cicchetti, D., & Balla, D. (2005). Vineland adapt-
Identification of autism spectrum disorders using the Child ive behavior scales (2nd ed.). Circle Pines, MN: American
Behavior Checklist in Singapore. Journal of Autism and Guidance Service.
Developmental Disorders, 41(9), 1147–1156. Spence, S.H. (1998). A measure of anxiety symptoms among
Ooi, Y.P., Rescorla, L., Sung, M., Fung, D.S., Woo, B., & Ang, children. Behaviour Research and Therapy, 36(5), 545–566.
R.P. (2014). Comparisons between autism spectrum disor- Swets, J.A. (1988). Measuring the accuracy of diagnostic sys-
ders and anxiety disorders: Findings from a clinic sample in tems. Science, 240(4857), 1285–1293.
Singapore. Asia-Pacific Psychiatry, 6(1), 46–53.
Rescorla, L., Kim, Y.A., & Oh, K.J. (2014). Screening for ASD
with the Korean CBCL/11=2–5. Journal of Autism and Devel- Supporting Information
opmental Disorders, Advance online publication.
Rutter, M., Bailey, A., & Lord, C. (2003). Social Communication Additional Supporting Information may be found in the
Questionnaire. Los Angeles, CA: Western Psychological Services. online version of this article.

42 Alexandra Havdahl et al./Utility of the CBCL in screening for ASD INSAR