Adults - Aseba

This excerpt is taken from:Manual for theASEBA AdultForms & ProfilesFor Ages 18-59• Adult Self-Report• Adult Behavior ChecklistAn Integrated System ofMulti-informant AssessmentThomas M. Achenbach, University of Vermont& Leslie A. Rescorla, Bryn Mawr College

Chapter 9Reliability, Internal Consistency, Cross-InformantAgreement, and StabilityReliability refers to agreement between repeatedassessments of characteristics when the characteristicsthemselves are expected to remain constant. Wheninstruments such as the ASR and ABCL are completed,it is important to know the degree to which scale scoresremain consistent over periods when the adaptive characteristicsand problems of the people who are assessedare not likely to change much. In this chapter,we present test-retest reliabilities for the ASR andABCL scale scores over 1-week intervals.Another property of scale scores is their internalconsistency. This refers to the degree to which theitems of a scale are correlated with each other. Internalconsistency is sometimes called split-half reliability,because it can be estimated by correlating the sumof scores on half the items of a scale with the sum ofscores on the other half. However, internal consistencyamong the items of a scale scored on a single occasioncannot tell us the reliability with which the scalewill produce the same results on different occasions.Although test-retest reliability and internal consistencyare typically viewed as psychometric propertiesof the scales themselves, reports of adaptive functioningand problems inevitably depend on the informants’perspectives. Assessment of adults has traditionallyrelied primarily on the adults’ self-reports via interviews,tests, and questionnaires. Although it has longbeen clear that multi-informant reports are needed forassessing children and adolescents, the need for multiinformantassessment of adults is becoming evident,as well (Klonsky et al. 2002; Meyer, 2002; Meyer etal., 2001). As illustrated in preceding chapters, theASR and ABCL are designed to obtain and systematicallycompare data from self-reports and reportsby other people for the assessment of individuals andfor research. In this chapter, we will present findingsfor cross-informant agreement between scale scoresobtained from self-reports and reports by others.91A further aspect of scale scores is their stabilitywhen the same informants complete forms over intervalslong enough for behavior to change. Stability canbe affected by many factors, including aging, therapeuticinterventions, and important life events such asmarriage, job changes, and traumas. Nevertheless, dataon the stability of scale scores for substantial samplesof adults can provide reference points for the typicallevel of those scales’ stability.TEST-RETEST RELIABILITIESOF SCALE SCORESTo assess reliability in both the rank ordering andmagnitude of scale scores, we computed test-retestPearson correlations (r) and t tests of differences betweenASR ratings and between ABCL ratings on twooccasions. The test-retest reliability samples wereobtained by requesting participants in our 1999 NationalSurvey to complete their respective forms twiceat intervals averaging 7 days. Because the ASR forages 30-59 used in the 1999 National Survey did notinclude the Education items, we report the test-retestr that was reported for the Education scale in theManual for the YASR and YABCL (Achenbach, 1997).The items and scoring were the same as for the currentEducation scale. As detailed in the Manual for theYASR and YABCL, the data were obtained from 232young adults who completed the YASR twice at a meanof 7 days. They included 50 participants in our previousnational sample, 11 participants in a follow-up ofadolescent mental health services, 123 American students,and 48 Turkish students who completed a Turkishtranslation of the ASR.Test-Retest CorrelationsAs shown in Table 9-1, reliability was generallyvery high, with all test-retest rs being significant atp

929. Reliability, Consistency, Agreement, and StabilityTable 9-1One-Week Test-Retest Reliabilities, Alphas, and Cross-Informant Correlations aTest-Retest r Alpha Coefficients Cross-Informant rScales ASR ABCL ASR ABCL ASR x ABCLAdaptive Functioning N = 51 54 295 402 1,196Friends .82 .82 .69 .67 .48Spouse/Partner .85 .84 .78 .76 .54Family b .74 NA f NA NAJob b .71 NA .60 NA NAEducation b .80 NA .51 NA NAMean Adaptive b .79 NA f NA NASubstance UseTobacco .93 .94 f f .79Alcohol .82 .92 f f .67Drugs .99 .87 f f .42Mean Substance Use .96 .91 f f .69Critical Items .87 d .84 .74 .81 .43Empirically BasedAnxious/Depressed .87 d .75 .88 .90 .44Withdrawn .87 d, e .73 .78 .81 .31Somatic Complaints .78 d, e .88 d, e .82 .77 .36Thought Problems .91 .86 .51 .71 .30Attention Problems .91 d .87 .87 .88 .38Aggressive Behavior .87 d .87 .83 .91 .42Rule-Breaking Behavior .79 d .86 .86 .86 .46Intrusive .83 d, e .88 .72 .70 .38Internalizing .89 d .80 .93 .92 .43Externalizing .91 d .92 .89 .93 .44Total Problems .94 d .92 .97 .97 .42Mean r c and mean alpha .88 c .86 c .83 .85 .40 cDSM-OrientedDepressive Problems .86 d, e .83 d, e .82 .88 .43Anxiety Problems .86 d .77 .68 .70 .41Somatic Problems .77 .87 d, e .79 .75 .34Avoidant Personality Problems .85 d .75 .74 .77 .35AD/H Problems .84 d .89 .84 .83 .35Inattention .80 d .81 .79 .75 .36Hyperactivity-Impulsivity .81 .91 .76 .76 .34Antisocial Personality Problems .84 d .87 .79 .87 .42Mean r c and mean alpha .83 c .85 c .78 .79 .38 cMean Q correlation between problem items NA NA NA NA .30aTest-retest samples were 1999 National Survey respondents interviewed at mean intervals of 7 days, exceptEducation scale for which data were obtained from 232 participants in previous national survey and other sources(Achenbach, 1997) described in the text. Cronbach’s alpha was computed for referred and nonreferred samples ofages 30-59 described in Chapter 10, except Education which was computed for age 18-29 YASRs (N=305). Crossinformantrs were for National Survey respondents. All rs were p Time2, at p

9. Reliability, Consistency, Agreement, and Stability 93r for the empirically based problem scales was .88 onthe ASR and .86 on the ABCL, while the rs for TotalProblems were .94 and .92, respectively. For theDSM-oriented scales, the mean r was .83 on the ASRand .85 on the ABCL.There were significant (p

949. Reliability, Consistency, Agreement, and Stability.60 to .78, except the Education scale, whose alphawas .51. These alphas are reasonable for scales thathave relatively few items.For the empirically based problem scales, the alphasranged from .51 to .97. The only alpha

9. Reliability, Consistency, Agreement, and Stability 95Table 9-2Long-Term Stabilities of Empirically Based Problem ScalesMean of YASRScales YASR YABCL & YABCL r bN = 484 N = 587Anxious/Depressed .63 .66 a .65Withdrawn .54 .57 a .56Somatic Complaints .54 .36 .46Thought Problems .42 .30 .36Attention Problems .54 a .67 a .61Aggressive Behavior .61 a .68 .65Rule-Breaking Behavior .56 .57 .57Intrusive .57 a .61 a .59Internalizing .62 .67 a .65Externalizing .63 a .70 .67Total Problems .65 a .68 .67Mean r b .58 .60 .59Note. Mean intervals were 39 months for YASRs and 44 months for YABCLs. All Pearson rs were significantat p Time 2, p

Chapter 10ValidityValidity refers to the accuracy with which instrumentsassess what they are supposed to assess.ASEBA instruments serve many purposes, and theirvalidity can be evaluated in multiple ways. A fundamentalpurpose of the ASR and ABCL is to aid in identifyingneeds for help with behavioral, emotional, and socialproblems and adaptive functioning. The ASR andABCL also provide well-differentiated pictures ofpeople’s functioning in terms of specific problems andadaptive characteristics, aggregations of related problemsinto empirically based and DSM-oriented scales,and broader aggregations of items that encompass diverseaspects of functioning. In this chapter, we presentevidence for the content validity, criterion-relatedvalidity, and construct validity of the ASR and ABCL.CONTENT VALIDITYThe most basic kind of validity is content validity,which is the degree to which an instrument’s contentincludes what the instrument is intended to assess.Problem ItemsThe ASR and ABCL problem items are productsof a long process of development, testing, and refinementon the basis of research and practical experience.The process began with the selection of itemsfor assessing children and youth on the basis of extensiveliterature searches, consultation with relevant professionals,and repeated pilot testing in a variety ofsamples (Achenbach, 1965, 1966; Achenbach &Edelbrock, 1983; Achenbach & Lewis, 1971). Applicationsof the ASEBA approach to assessment ofadults began in the 1980s with development of theYASR and YABCL to assess adults who had previouslybeen assessed with ASEBA school-age instruments(Achenbach, 1991a; Achenbach, Howell,McConaughy, & Stanger, 1995c; Stanger, MacDonald,McConaughy, & Achenbach, 1996).The Manual for the Young Adult Self-Report andYoung Adult Behavior Checklist (Achenbach, 1997)provides details of the refinement and testing of theitems and scales for the young adult forms and profiles.The development of the forms for ages 30-59began in the 1990s with the addition of items to theyoung adult forms, further testing and revision, anduse of the forms for ages 30-59 in our 1999 NationalSurvey and in various clinical and nonclinical settings.As described in Chapter 7 of this Manual, our separatefactor analyses of the forms for ages 18-29 andfor ages 30-59 yielded such similar results that thedata from both sets of forms were combined in thefinal analyses from which the ASR and ABCL syndromeswere derived.As documented in Chapter 12, the current ASRand ABCL syndromes are quite similar to those derivedfrom the YASR and YABCL (Achenbach, 1997)via different factor analytic methodology, and scoreson the current syndromes correlate highly with scoreson the 1997 versions. All problem items of the currentversions of the adult forms were (a) scored significantlyhigher on one or both forms for adults referredfor mental health or substance use services than fordemographically similar adults who had not been referredfor such services in the preceding 12 months(details in Chapter 11), and/or (b) loaded significantlyon empirically based syndromes (details in Chapter7), and/or (c) were identified by the expert panel asbeing very consistent with DSM-IV diagnostic categories(details in Chapter 4).Adaptive Functioning ItemsThe adaptive functioning items were hypothesizedto reflect aspects of functioning that are important forsuccessful adaptation in various areas. Most of theitems were found to discriminate significantly betweenreferred and nonreferred young adults (Achenbach,96

10. Validity 971997). A few items have been made more differentiatedand a few items have been added since the 1997version of the YASR, as detailed in Chapter 12. TheASR adaptive functioning items pertaining to friendsand family are relevant to nearly all adults, althoughthe family items are not included on the ABCL, becausemany informants may not be knowledgeableabout them. The items pertaining to spouse/partner,job, and education are completed only for adults forwhom they have been relevant in the preceding 6months.Although included in the ASR Job section, item IV.E.I am satisfied with my job situation, is not scored onthe Job scale. It is not scored because it was unexpectedlyfound to be rated significantly higher by referredthan nonreferred respondents in the samples analyzedfor this Manual, whereas it was rated significantly higherby nonreferred than referred young adults in our previousanalyses of the YASR (Achenbach, 1997). Becauseof these contradictory findings, we decided to omit itemIV.E from the Job scale, although it may be clinicallyuseful. As detailed in Chapter 11, almost all other adaptivefunctioning items were rated significantly more favorablyfor nonreferred than referred adults on one orboth adult forms.In summary, the content validity of the ASR andABCL items has been supported by a long process ofitem development, testing, and revision, as well as byfindings that most of the items retained for scoring onscales discriminate significantly between demographicallysimilar referred and nonreferred adults.CRITERION-RELATED VALIDITYOF SCALE SCORESCriterion-related validity refers to the strengthof association between a particular measure, such asa scale scored from an ASEBA form, and an externalcriterion for characteristics that the scale is intendedto assess. In the preceding section, we mentioned thatmost of the items retained for ASR and ABCL scalesdiscriminated significantly between referred andnonreferred adults on one or both forms. Here wefocus on associations between scales comprising particularsets of ASEBA items and external criterion variables.We will first present new validity evidence basedon analyses done for this Manual. We will then summarizevalidity evidence from other sources.Demographically Similar Samples ofReferred and Nonreferred <strong>Adults</strong>To test the ability of each ASEBA scale to discriminatebetween referred and nonreferred adults, weconstructed referred and nonreferred samples thatwere matched for gender and were similar in age distributions.The referred adults came from 17 mentalhealth and substance abuse treatment settings. In addition,to augment the sample of ABCLs, we usedABCLs that were completed for participants in our1999 National Survey who reported receiving mentalhealth or substance abuse services in the preceding12 months. The ASRs and ABCLs for nonreferredadults were for participants in our 1999 National Surveywho reported that they had not received mentalhealth or substance abuse services in the preceding12 months. We controlled for differences in age andethnicity by treating them as covariates in ANCOVAsand as independent variables in multiple regressionanalyses. Table 10-1 summarizes characteristics of thereferred and nonreferred samples.Multiple Regression Analyses ofAdaptive Functioning ScalesTo test the associations of referral status and demographicvariables with scale scores, we used a structuralequation modeling (SEM) approach whereby weregressed the raw scores for a scale (the dependentvariable) on the independent variables of referral status,gender, age (dichotomized as 18-35 vs. 36-59),and nonLatino white vs. other ethnicity. We enteredall independent variables simultaneously to test thepredictive power of each independent variable withthe others partialed out.For each adaptive scale, Table 10-2 displays thepercentage of variance uniquely accounted for by referralstatus, with the effects of age, gender, andethnicity partialed out. Cohen’s (1988) criteria for effectsizes (ES) in multiple regression are as follows:Small = 2-13%; medium = 13-26%; and large >26%.The ES for referral status were large for 5 of the 6

9810. ValidityTable 10-1Characteristics of Demographically Similar Referred vs. Nonreferred SamplesASRABCLCharacteristics Ref. Nonref. Ref. Nonref.N = 846 N = 846 N = 447 N = 447GenderMen 45% 45% 53% 53%Women 55% 55% 47% 47%AgeMean 26.1 29.9 30.6 31.9SD 8.9 9.5 12.8 11.9Education aMean 4.2 4.0 3.7 3.9SD 1.8 1.9 1.9 1.9SES b Mean 1.7 2.0 1.7 1.9SD 0.7 0.7 0.8 0.7EthnicityNonLatino White 92% 66% 84% 69%African American 3% 17% 7% 16%Latino 2% 11% 5% 11%Mixed or Other 4% 7% 4% 5%aEducation was scored: 1 = No high school diploma or General Equivalency Diploma (GED); 2 = GED; 3 = Highschool graduate; 4 = Some college; 5 = Associate’s degree; 6 = Bachelor’s or RN degree; 7 = Some graduateschool; 8 = Master’s degree; 9 = Doctoral or Law degree.bSES was scored 1 = lower, 2 = middle, 3 = upper, based on an updated version of Hollingshead’s (1975) 9-step scalefor the occupation of the spouse/partner holding the higher status job: Hollingshead scores 1.0-3.9 = lower; 4.0-6.9= middle; 7.0-9.0 = upper; we assigned 2-digit codes because occupations that were not clearly scorable were giventhe mean of their most likely scores.ASR scales and were small for the ASR Family scaleand for both ABCL adaptive functioning scales. Thus,after partialing out demographic variance, referral statusaccounted for significant variance in all the adaptivescales of the ASR. Figure 10-1 graphically displaysthe mean scores on each adaptive scale.Demographic Effects. Scores were significantlyhigher for ages 36-59 than 18-35 on the ASR Joband Mean Adaptive scales, but, as Table 10-2 shows,the effects were small according to Cohen’s (1988)criteria. On the ABCL Friends scale, women obtainedsignificantly higher scores than men, while on the ASRFriends scale, nonwhites scored themselves significantlyhigher than whites. However, these differencesdid not exceed chance expectations for the number ofanalyses (Sakoda et al., 1954).Multiple Regression Analyses ofProblem ScalesAs we did for the adaptive functioning scales, weused SEM to regress the raw scores of each problem

10. Validity 99Table 10-2Percent of Variance Accounted for by Significant (p

10. Validity 101Referred MenReferred WomenNonreferred MenNonreferred WomenFigure 10-1 (cont.). Mean scores for adaptive functioning scales.

10210. ValidityTable 10-3Percent of Variance Accounted for by Significant (p

10. Validity 103Referred MenReferred WomenNonreferred MenNonreferred WomenFigure 10-2. Mean scores for problem scales.

10410. ValidityReferred MenReferred WomenNonreferred MenNonreferred WomenFigure 10-2 (cont.). Mean scores for problem scales.

10. Validity 105Referred MenReferred WomenNonreferred MenNonreferred WomenFigure 10-2 (cont.). Mean scores for problem scales.






10. Validity 111egory, we have identified a borderline clinical rangefor each scale. The inclusion of a borderline categoryimproves the basis for decisions about needs for help.As an example, a scale score in the borderline rangetells us that enough problems have been reported tobe of concern but not so many that a person clearlyneeds professional help. If a person obtains one ormore scale scores in the borderline range but none inthe clinical range, we should consider options such asthe following: (a) Obtain ratings from more informantsto determine whether they view the person as being inthe normal, borderline, or clinical range; (b) have theinitial informants rate the person again after 2 to 3months to see whether the borderline scores moveinto the normal or clinical range; (c) use additional assessmentprocedures to evaluate the kinds of problemscomprising the scales that reached the borderlinerange. In other words, borderline scores can help usersmake more differentiated decisions than if all scoresmust be categorized as normal vs. clinical.Continuous, quantitative scale scores afford greaterstatistical power than categorization of scores into acouple of levels, such as normal vs. borderline andclinical. Nevertheless, users may wish to distinguishdichotomously between normal and deviant scalescores. In the following sections, we report findingsthat indicate the degree to which dichotomous classificationsof ASEBA scale scores according to the normalrange vs. combined borderline and clinical rangesdistinguish between demographically similarnonreferred vs. referred people. Because the borderlinerange encompasses scores that are high enough tobe of concern, we have included it with the clinicalrange for our dichotomous comparisons of deviantscores with scores that are in the normal range.Odds Ratios (ORs)One approach to analyzing associations betweentwo dichotomous ways of classifying people is by computingthe relative risk odds ratio (OR; Fleiss, 1981),which is used in epidemiological research. The ORindicates the odds that people who have a particularrisk factor also have a particular condition (usually adisorder), relative to the odds that people who lackthe risk factor have the condition. The comparisonbetween outcome rates for those who do vs. do nothave the risk factor is expressed as the ratio of theodds of having the outcome if the risk factor is present,to the odds of having the outcome if the risk factor isabsent. For example, a study of relations betweensmoking (the risk factor) and lung cancer (the outcome)may yield a relative risk OR of 6. This meansthat people who smoke have 6 times greater odds ofdeveloping lung cancer than people who do not smoke.We applied OR analyses to the relations betweenASEBA scale scores and referral status as follows:For each ASEBA scale, we first classified people fromour referred and nonreferred samples according towhether their scores were in the normal range or weredeviant (i.e., were in the borderline or clinical range).Deviant scores were thus equivalent to a “risk factor”in epidemiological research, whereas referral vs.nonreferral was the outcome. We then computed theodds that people whose scores were deviant on aparticular scale were from the referred sample, relativeto the odds for people whose scores were notdeviant on that scale.The OR is a nonparametric statistic computed froma 2 x 2 table. For the analysis of each scale scoredfrom each form, we therefore included both gendersand all ages to provide a summary OR across all groupsfor whom the form was scored. The statistical significanceof the OR is evaluated by computing confidenceintervals.Adaptive Functioning Scales. Table 10-4 displaysthe ORs for relations between deviant scoresand referral status for the ASR and ABCL adaptivefunctioning scales. Table 10-4 also shows the percentof referred people whose scores were deviant accordingto the cutpoints for the normal vs. combined borderlineand clinical ranges on each scale. For all ASRadaptive functioning scales, confidence intervalsshowed that the ORs were significantly (p

11210. ValidityTable 10-4Significant (p

10. Validity 113Table 10-5Significant (p 1 syndrome in deviant range 3 3 59 53 32 28Int and/or Ext in deviant range 4 3 57 52 25 26Mean Adaptive and/or TotalProblems in deviant range 4 NA 57 NA 25 NACritical Items 9 4 41 23 7 8DSM-OrientedDepressive Problems 7 4 38 24 8 8Anxiety Problems 3 2 b 14 13 b 6 7Somatic Problems 3 2 b 20 15 8 9Avoidant Personality Problems 4 3 22 20 6 8AD/H Problems 6 3 29 18 6 8Inattention 6 3 28 19 6 8Hyperactivity-Impulsivity 4 2 b 22 14 b 7 8Antisocial Personality Problems 6 5 32 27 7 7> 1 DSM scale in deviant range 4 3 59 52 27 26Substance UseTobacco 3 4 20 32 8 11Alcohol 2 b 3 14 b 17 8 7Drug 6 7 28 24 6 7Mean Substance 5 5 29 27 7 7Note. Odds ratios indicate the odds that referred adults obtained scores in the combined borderline and clinicalranges. On all scales, the proportion of referred adults scoring in the clinical range significantly exceeded theproportion of nonreferred adults at p

11410. ValidityCross-Validated Correction forShrinkageTo correct for shrinkage, we used a “jackknife”procedure whereby the discriminant function for eachsample was computed multiple times with a differentperson held out of the sample each time (SAS Institute,1999). Each discriminant function was then crossvalidatedby testing the accuracy of its prediction foreach of the “hold-out” people. Finally, the percentageof correct predictions was computed across all thehold-out people. It is these cross-validated predictionsthat we will present.Cross-Validated Percent of PeopleCorrectly ClassifiedTable 10-6 displays the cross-validated percent ofpeople who were correctly classified by the discriminantanalyses using the three different sets of candidatepredictors for each instrument. As you can see inTable 10-6, the three sets of predictors achieved totalaccuracies ranging from 71% (syndromes) to 87%(problem items) for the ASR. For the ABCL, the totalaccuracies were quite similar for the three sets of predictors,ranging from 65% for the problem items to68% for the syndrome scales and for the DSM-orientedscales. By looking again at Table 10-6, you cansee that the ASR problem items produced the bestresults for each of the four classification parameters,as follows: True positives (sensitivity) = 80%; truenegatives (specificity) = 95%; false positives = 20%;false negatives = 5%.Results for Specific Scales. In the discriminantanalyses that used the eight syndrome scales as candidatepredictors, the following syndromes survived assignificant predictors for both the ASR and ABCL:Anxious/Depressed; Thought Problems; and Rule-Breaking Behavior. For the ASR, the Attention Problemssyndrome and, for the ABCL, the Intrusive syndromealso survived. Among the syndromes that survivedas significant predictors for both the ASR andABCL, one was from the Internalizing grouping, onewas from the Externalizing grouping, and one was asyndrome that was not strongly associated with eitherthe Internalizing or Externalizing grouping. These findingsindicate that no one pattern of problems outweighsall others as a significant predictor of need for services.From the six DSM-oriented scales, the followingfour survived as significant predictors for both theASR and ABCL: Depressive Problems; AnxietyProblems; Avoidant Personality Problems; and AntisocialPersonality Problems. Based on both the syndromesand the DSM-oriented scales, it is thus evidentthat diverse sets of problems contribute significantlyto discriminating between people who aredeemed to need professional help and those who arenot.Results for Specific Problem Items. For theASR, 33 items survived as significant predictors, whilefor the ABCL, 14 survived. In both the ASR and ABCLanalyses, the first item to be entered was 103. Unhappy,sad, or depressed. Item 103 has been repeatedlyfound to be an especially powerful discriminatorbetween referred vs. nonreferred children and adolescentsaccording to different informants in multiplesamples (Achenbach, 1991b,d; Achenbach &Edelbrock, 1983, 1986, 1987; Achenbach &Rescorla, 2001; Verhulst, Akkerhuis, & Althaus,1985). Even when entered with a very large numberof other predictors, this item often obtains considerablylarger discriminant function coefficients than theother significant predictors. As shown in Chapter 11,ANCOVAs of all problem items revealed that item103 was more strongly associated with referral statusthan any other problem item on the ASR (ES = 27%)and ABCL (ES = 11%).For both the ASR and ABCL, the second item tobe entered was 6. Uses drugs (other than alcoholor nicotine) for nonmedical purposes. The survivalof this item as a strong predictor on both the ASR andABCL indicates that drug use is likely to be associatedwith a substantial proportion of referrals for services.The only other item to survive as a significant predictorin the discriminant functions for both the ASRand ABCL was 47. Lacks self-confidence.In summary, the discriminant analyses achieved thebest cross-validated accuracy of 87% of participants

10. Validity 115Table 10-6Cross-Validated Percent of <strong>Adults</strong> Correctly Classified as Referredvs. Nonreferred by Discriminant AnalysesASRABCL% Correctly % CorrectlyCandidate Predictors Classified TP TN FP FN Classified TP TN FP FN8 Syndromes 71 65 78 22 35 68 56 80 20 446 DSM- oriented scales 77 72 82 18 28 68 57 79 21 43All problem items 87 80 95 5 20 65 55 76 24 45Note. TP = true positives (sensitivity); TN = true negatives (specificity); FP = false positives; FN = false negatives.correctly classified when selecting from among all theproblem items on the ASR. The priority of item 103.Unhappy, sad, or depressed as a predictor on boththe ASR and the ABCL and its strong discriminativepower in analyses of numerous samples of childrenand adolescents attest to its association with diverseconditions that warrant professional help across a wideage span.PROBABILITY OF PARTICULARTOTAL PROBLEMS SCORESBEING FROM THE REFERREDVS. NONREFERRED SAMPLESTo provide further perspectives on relations betweenASEBA scores and referral status, Table 10-7displays the probabilities that particular T scores werefrom referred samples rather than from demographicallysimilar nonreferred samples. The probabilitieswere determined by tabulating the proportion ofpeople from our matched referred and nonreferredsamples within each of the T score intervals shown inTable 10-7. We used T scores in order to provide auniform metric across all gender/age groups on theASR and ABCL.The probability that a score was from the referredsample increased fairly steadily as the Total Problemsscores increased. Users can consult Table 10-7 toestimate the probability that particular Total Problemsscores represent deviance severe enough to warrantconcern.CONSTRUCT VALIDITY OFASEBA SCALESAccording to a dictionary definition, a construct is“an object of thought constituted by the ordering orsystematic uniting of experiential elements” (Gove,1971, p. 489). ASEBA scales can be viewed as representingconstructs that have been derived by systematicallyordering scores on the items of the ASEBAforms, which tap people’s experience pertaining to theindividual being assessed.Construct validity concerns evidence that supportshypothesized variables (hypothetical constructs)for which there are no definitive criterion measures.A primary reason for developing ASEBA instrumentswas to derive syndromal constructs thatembody patterns of problems that occur together. Studiesof ASEBA child and adolescent syndromes haverevealed numerous correlates and considerable developmentalstability for the syndromes (evidence hasbeen reviewed by Achenbach & Rescorla, 2000,2001).The correlates and developmental courses of thedifferent syndromes indicate that they reflect importantdifferences in patterns of child and adolescent functioning.The validity of constructs such as the ASEBAsyndromes is supported by the accumulation of evidencefor systematic relations between measures ofthe constructs and other variables. These systematicrelations are called nomological (i.e., lawful) networks(Cronbach & Meehl, 1955).

11610. ValidityTable 10-7Probability of Total Problems T Scores Being from Referred SamplesTotal ProblemsT Scores ASR ABCLN = 1,692 N = 8940-35 .27 .1936-39 .20 .3340-43 .30 .1544-47 .37 .3548-51 .40 .5352-55 .46 .4556-59 .50 .5660 a -63 .60 .6764-67 .69 .6868-100 .87 .72Note. Samples were demographically similar referred and nonreferred adults.aT scores > 60 are in the combined borderline and clinical range.Each ASEBA syndrome can be viewed in statisticalterms as representing a latent variable derivedby factor analyzing ASEBA items. The versions of asyndrome derived from separate factor analyses ofthe ASR and ABCL provide different ways of operationalizingthe construct represented by the syndrome.Furthermore, the versions of a syndrome scored fromASR and ABCL ratings provide different quantitativemeasures of the latent variables represented by thesyndromes.People differ in their knowledge of an individual’sfunctioning, in their roles in relation to the individualbeing assessed, in what they remember, and in personalcharacteristics that can affect their ratings. Consequently,the correlations among ratings by differentrespondents, especially those playing different roleswith respect to the individual they rate, may be modest,as shown in Chapter 9. Nevertheless, the testretestreliability of ASR and ABCL ratings is good, asdocumented in Chapter 9. Furthermore, the contentand criterion-related validity of ratings by different informantshas been documented in the preceding sectionsof this chapter. The findings thus indicate thatdifferent informants can contribute to the assessmentprocess.Assessment of the syndromal constructs via datafrom multiple sources is consistent with the way in whichpsychological constructs are conceptualized and evaluated.Because psychological constructs involve inferencesabout latent variables that are not directly observable,their validity must be evaluated in terms ofvarious kinds of indirect evidence relevant to their validity.The Bibliography of Published Studies UsingASEBA Instruments (Bérubé & Achenbach, 2003)lists some 4,500 published studies of ASEBA instruments.Many of the studies provide evidence for theconstruct validity of ASEBA scales in terms of significantassociations with other variables, prediction andevaluation of outcomes, and consistency with theoreticalformulations. In the following sections, we summarizeseveral kinds of support for the construct validityof ASEBA adult scales. Although some of thefindings were obtained with earlier versions of thescales, the earlier versions correlate highly with the2003 versions, as documented in Chapter 12.Correlations of ASR Scales withSCL-90-R ScalesThe Symptom Checklist-90-Revised (SCL-90-R;Derogatis, 1994) includes 90 self-report problem items

10. Validity 117that are rated on 5-point scales ranging from 0 = Notat all to 4 = Extremely. Respondents rate each itemaccording to “how much the problem has distressed orbothered you during the past 7 days including today.”Items were chosen to assess the following nine symptomconstructs: Somatization, Obsessive-Compulsive,Interpersonal Sensitivity, Depression, Anxiety, Hostility,Phobic Anxiety, Paranoid Ideation, andPsychoticism. Each construct is scored by summingthe 0 to 4 ratings on the items measuring the construct.The items of the SCL-90-R are also scored in terms ofthe following three global indices: The Global SeverityIndex (GSI), which is computed by summing the ratingson all items and dividing by the number of itemsthat were rated; the Positive Symptom Total (PST),which is the number of items that were endorsed; andthe Positive Symptom Distress Index, which is calculatedby dividing the sum of all 1-4 ratings by the PST.Like the ASR, the SCL-90-R can be self-administeredunder many conditions in about 15 to 20 minutes.Unlike the ASR, the SCL-90-R does not haveadaptive functioning items, open-ended items that inviterespondents to provide details, substance usescales, DSM-oriented scales, or the kinds of itemsthat comprise the ASR’s Rule-Breaking Behavior andIntrusive syndromes. However, because the SCL-90-R and ASR are apt to be candidates for use undersimilar clinical and research conditions, it is helpful toknow the degree to which their scales tap similar constructs.Table 10-8 displays Pearson rs between ASRproblem scales and SCL-90-R scales scored fromforms completed by 66 clients of a research clinic fortreatment of substance abuse in Vermont. (The SCL-90-R PST global index is not shown, because its functionis similar to that of the other SCL-90-R globalindices, both of which correlated higher than the PSTwith the ASR scales.)There were significant correlations between everyASR problem scale and every SCL-90-R scale shownin Table 10-8. Most of the correlations met Cohen’s(1988) criteria for large ES (r >.50). Twenty of the rswere >.70. There were thus strong and pervasive associationsbetween scores on the two instruments.Among the ASR syndromes, the Anxious/Depressedsyndrome had its highest rs (.69 to .78) with the SCL-90-R Obsessive-Compulsive, Interpersonal Sensitivity,Depression, and Anxiety dimensions. The ASR SomaticComplaints syndrome had its highest r (.66) withthe Somatization dimension. The ASR Thought Problemssyndrome had its highest rs (both .53) with theObsessive-Compulsive and Psychoticism dimensions.The ASR Attention Problems syndrome had its highestr (.70) with the Obsessive-Compulsive dimension. Andthe Aggressive Behavior syndrome had its highest r (.69)with the Hostility dimension. Reflecting the lack of correspondingSCL-90-R items, the highest rs of the ASRRule-Breaking Behavior and Intrusive syndromesreached only .43 and .41, respectively, with an SCL-90-R dimension. However, the Rule-Breaking Behaviorsyndrome correlated .49 with the SCL-90-R PositiveSymptom Total. Reflecting the predominance of internalizingsymptoms on the SCL-90-R, the ASR Internalizingscale had rs in the .60s and .70s with 8 of the9 SCL-90-R symptom dimensions, the Global SeverityIndex, and the Positive Symptom Total.Among the ASR DSM-oriented scales, rs in the .70swere found for the Depressive Problems scale with theSCL-90-R Depression dimension, for the AvoidantPersonality Problems scale with the SCL-90-R InterpersonalSensitivity dimension, and for the AttentionDeficit/Hyperactivity Problems scale with the SCL-90-R Obsessive-Compulsive scale. The fact that everySCL-90-R scale correlated >.62 with one or more ASRscales indicates that the types of problems assessed bythe SCL-90-R are also assessed by the ASR.Longitudinal Relations BetweenPre-Adult Scales and Adult ScalesAmerican National Sample: 3-Year PredictiveRelations. Longitudinal studies have demonstratedsignificant predictive correlations from scores on theYSR and CBCL to scores on YASR and YABCLscales that correlate highly with the 2003 adult scales.In an American national sample, 3-year longitudinal rsfrom syndrome scores on YSRs completed by adolescentsto syndrome scores on YASRs completed bythe same participants 3 years later ranged from .34for 367 females on the Somatic Complaints syndrometo .54 for 340 males on the Anxious/Depressed syndrome(Achenbach et al., 1995c).

11810. ValidityNote. Participants were 66 adults enrolled in a substance abuse treatment program. SCL-90-R Symptom Distress Index is not listed, because itssignificant correlations with ASR scales were all lower than the corresponding SCL-90-R Severity Index and Positive Symptom Total correlations withASR scales.SCL-90-R ScalesSomati- Obsess.- Interper. Depres- Phobic Paranoid Psychot- Global SymptomASR Scales zation Compuls. Sensitivity sion Anxiety Hostility Anx. Ideation icism Severity TotalEmpirically BasedAnx./Dep. .57 .71 .78 .76 .69 .47 .65 .62 .68 .76 .68Withdrawn .42 .60 .60 .46 .44 — .46 .49 .57 .55 .57Somatic .66 .62 .57 .62 .58 — .58 .48 .53 .66 .60Thought .49 .53 .49 .41 .49 .35 a .52 .40 .53 .53 .48Attention .46 .70 .60 .62 .58 .42 .61 .48 .60 .65 .64Aggressive .41 .58 .67 .63 .53 .69 .45 .50 .55 .63 .57Rule-Break. — .43 — .36 a — — — — .39 .39 .49Intrusive — .35 a — — — .41 — — — — —Internal. .64 .75 .77 .75 .69 .46 .67 .62 .70 .78 .72External. .37 a .58 .57 .55 .49 .58 .41 .40 .51 .57 .57Total .56 .74 .72 .70 .65 .53 .62 .57 .67 .74 .70DSM-OrientedDepressive .54 .69 .69 .77 .65 .42 .61 .54 .61 .73 .62Anxiety .54 .62 .66 .64 .64 — .60 .53 .59 .67 .62Somatic .63 .59 .53 .57 .53 .38 a .53 .45 .50 .62 .56Avoidant .51 .63 .70 .53 .52 .39 a .53 .56 .60 .63 .66AD/H .52 .71 .61 .58 .59 .45 .59 .43 .58 .65 .63Antisocial — .55 .52 .49 .45 .52 .40 .41 .52 .53 .57Table 10-8Significant (p

10. Validity 119Also in the American national sample, scores onCBCL syndromes yielded 3-year longitudinal rs withYABCL syndromes that ranged from .40 for the SomaticComplaints syndrome among 377 females to.67 for the Anxious/Depressed syndrome among 366males. Including Internalizing, Externalizing, and TotalProblems, the mean 3-year r from the CBCLscores to the YABCL scores was .59, which wasvery similar to the mean r of .60 between YABCLscales rated twice over 44 months by parents in thesame national sample, as was shown in Table 9-2.These rs were somewhat higher than the YSR toYASR rs, but the pattern of lowest and highest rswas the same as in the self-ratings. The overall meanr of .58 between all the YASR scales rated twiceover 39 months shown in Table 9-2 was also higherthan the mean YSR to YASR r of .48. The youngerages of the participants when they completed theYSR may explain the somewhat lower rs than werefound for the 39-month stability of the YASR. In anyevent, the similarity of the predictive rs between ratingsof adolescents on the YSR and CBCL to youngadult YASR/YABCL ratings at two points in adulthoodindicate that the YASR/YABCL scales tappedthe same constructs as the YSR and CBCL scales.An additional finding was that certain scores hadstrong associations with adult signs of disturbance,including police contacts, suicidal behavior, and druguse (Achenbach, Howell, McConaughy, & Stanger,1998).Dutch General Population Sample: 4-YearPredictive Relations. As part of a longitudinal studyof randomly selected Dutch children, 364 adolescentscompleted YSRs at ages 15 to 18 (Ferdinand, Verhulst,& Wiznitzer, 1995). Two years later, they completedYASRs, and did so again 2 years after that. Over the4-year interval from the YSR assessment to the secondYASR assessment, the mean r between YSR andYASR Total Problems scores was .49, compared toa mean r of .55 found over a 3-year interval in anAmerican national sample (Achenbach et al., 1995c).Allowing for the 1-year longer interval in the Dutchsample, the predictive relations from YSR to YASRTotal Problems scores were thus very similar forAmerican and Dutch young people.Ferdinand et al. also classified participants accordingto whether they scored >90 th percentile or

12010. Validitythe YSR Aggressive syndrome is viewed as the adolescentcounterpart of both these adult syndromes).The only exceptions to the findings of predictive specificityover 10 years were for the males’ scores on theThought Problems and Attention Problems syndromeson the YASR, which were most strongly predicted bythe YSR Anxious/Depressed and Thought Problemssyndromes, respectively. The YASR Internalizing,Externalizing, and Total Problems scores were alsomost strongly predicted by their YSR counterparts.An additional finding was that significantly more participantswhose YSR Total Problems scores were inthe clinical range met criteria for DSM-IV diagnosesin structured interviews 10 years later than participantswhose YSR Total Problems scores were not in theclinical range. Several YSR syndrome scales also predictedparticular kinds of DSM-IV diagnoses over the10-year period.Prediction of Adult SyndromeScores from Child/AdolescentASEBA Scores in Clinical SamplesAmerican Clinical Sample: 8-Year PredictiveRelations. The foregoing studies supported the constructvalidity of the adult scales in terms of predictiverelations from adolescent YSR and CBCL scores toadult YASR and YABCL scores over periods of 3 to10 years in American and Dutch samples that wererandomly selected to be representative of their respectivepopulations. The construct validity of the adult syndromeshas also been supported by predictive relationsfound for adults who had received mental healthservices when they were children or adolescents(Stanger, MacDonald, McConaughy, & Achenbach,1996). Structural equation modeling of the latent variablesjointly measured by YASR and YABCL syndromescores showed that CBCL syndrome scoresobtained at referral 8 years earlier were the most specificpredictors of their analogous adult syndrome constructs.In addition, the adult Intrusive syndrome (designatedas Shows Off in the Stanger et al. article) wasstrongly predicted by the CBCL Aggressive Behaviorsyndrome.The strong predictive relations between pre-adultscores on the Aggressive Behavior syndrome and adultscores on the Intrusive syndrome in the clinical follow-upsample are consistent with findings in theAmerican national sample (Achenbach et al., 1995c).Specifically, in longitudinal path analyses, the strongestpredictors of the mean of adult Intrusive syndromescores for 707 participants were adolescentAggressive Behavior syndrome scores, averagedacross CBCL and YSR ratings. This does not meantthat the Intrusive syndrome was merely the adult versionof the child/adolescent Aggressive Behavior syndrome,because the child/adolescent Aggressive Behaviorsyndrome was also the strongest predictor ofthe adult Aggressive Behavior syndrome.The findings thus indicate that the child/adolescentAggressive Behavior syndrome becomes differentiatedinto two syndromes on the adult instruments: One adultsyndrome, the Aggressive Behavior syndrome, comprisesovertly aggressive behaviors; the second adultsyndrome, designated as Intrusive, comprises behaviorsthat may be socially obnoxious but are not overtlyaggressive. According to these findings, during the transitionto adulthood, some aggressive adolescents mayretain the intrusive behaviors of the child/adolescentAggressive Behavior syndrome but manifest less overtaggressive behavior. Other aggressive adolescents,however, may continue to manifest the overtly aggressivebehavior that comprises the adult Aggressive Behaviorsyndrome.Dutch Clinical Sample: 10.5-Year PredictiveRelations. A similar study assessed Dutch adults whohad received mental health services at a mean of 10.5years earlier (Visser, van der Ende, Koot, & Verhulst,2000). In this study, scores for each YASR syndromewere regressed separately on initial scores for all eightCBCL syndromes and for all eight YSR syndromes.Scores for each YABCL syndrome were also regressedon each score for all eight CBCL syndromes.There were thus 24 separate regressions of adult syndromescores on childhood scores obtained at intakeinto mental health services (8 syndromes x 3 combinationsof respondents). Age, gender, SES, and lengthof follow-up interval were also included as candidatepredictors in the regression analyses. In 22 of the 24regression analyses, the adult syndromes were moststrongly predicted by their counterpart syndromes on

10. Validity 121the CBCL and YSR (with the childhood AggressiveBehavior syndrome being the counterpart of the adultIntrusive syndrome). The two exceptions were theYASR Somatic Complaints syndrome and the YASRThought Problems syndrome. These syndromes weremost strongly predicted by the YSR Anxious/Depressedand Social Problems syndromes, respectively.Relations of Adult Scales toDiagnostic AssessmentAmerican Clinical Sample. In a second followupof 181 of the Stanger et al. (1996) young adultswho had received mental health services as childrenor adolescents, DSM-III-R diagnoses were madeon the basis of structured diagnostic interviews(Achenbach, McConaughy, LaRiviere, & Stanger,1997). The interviewers also completed the GlobalAssessment of Functioning scale (GAF; AmericanPsychiatric Association, 1987) on the basis of the diagnosticinterviews and life history interviews with theparticipants. The participants completed YASRs eitherprior to the interviews or after the interviews, incounterbalanced order. Interviewers did not know theparticipants’ YASR scores.The YASR Total Problems score correlated .67with the number of DSM diagnoses and -.65 with GAFscores (negative r because high GAF scores indicatedgood functioning, whereas high YASR problem scoresindicated poor functioning). There were also many significantpoint-biserial rs between YASR syndromescales and DSM diagnoses, which were scored aspresent vs. absent. The highest point-biserial rs were.60, .47, and .46 between the YASR Anxious/Depressedsyndrome and diagnoses of Generalized AnxietyDisorder, Post-Traumatic Stress Disorder, and Agoraphobia/SocialPhobia, respectively; a point-biserialr of .53 between the YASR Aggressive Behavior syndromeand Oppositional Defiant Disorder (which wasincluded in the DSM diagnostic interview, even thoughit is often attributed only to pre-adults); and a pointbiserialr of .47 between the YASR Delinquent Behaviorsyndrome and Antisocial Personality Disorder.Although they were empirically derived from the “bottomup” by factor analysis, the YASR syndromes werethus significantly associated with certain DSM diagnosticconstructs, which were formulated using a “topdown”approach.American Community Sample. In an Americancommunity sample that had been studied longitudinallysince age 5, the Diagnostic Interview Schedule (DIS)was administered at age 18 years (Giaconia, Reinherz,Paradis, Hauf, & Stashwick, 2001). At age 21, theparticipants completed the YASR and their motherscompleted the YABCL. Analyses of age 21 outcomesfor diagnoses that had been made at age 18 showedthat participants who had been diagnosed as havingMajor Depressive Disorder at age 18 obtained significantlyhigher Internalizing scores on both the YASRand YABCL at age 21 than participants who had notreceived this diagnosis. Participants who had been diagnosedas having either Major Depressive Disorderor Drug Abuse/Dependence Disorder at age 18 obtainedsignificantly higher YASR Externalizing scoresat age 21 than participants who did not receive thesediagnoses.In the same sample, YASR Internalizing, Externalizing,and Total Problems scores were significantlyhigher for women who had been sexually abused beforeage 18 than for women who had not reportedsexual abuse (Silverman, Reinherz, & Giaconia, 1996).Some of these YASR scales were also significantlyhigher for men and women who had been physicallyabused than for those who had not reported physicalabuse.American Abused Sample. In a sample of adultswho reported being physically or sexually abused inchildhood, Posttraumatic Stress Disorder (PTSD) symptomscorrelated .86 with YASR Total Problems scoresand .80 with YASR Internalizing scores (Muller,Lemieux, & Sicoli, 2001). In addition, the Beck DepressionInventory (BDI) correlated .78 with YASR TotalProblems and .75 with YASR Internalizing. (Laumann-Billings and Emery, 2002, have reported r = .74 betweenthe BDI and Internalizing scored from the YASRand YSR.) The Beck Anxiety Inventory correlated .69and .63 with YASR Total Problems and Internalizing,respectively. Negative views of the self scored from theRelationship Scales Questionnaire correlated higher (r= .61) with YASR Internalizing scores than with eitherof the Beck scales or with PTSD symptoms.

12210. ValidityDutch General Population Sample. In a Dutchsample that had been randomly selected, adults whocompleted the YASR were interviewed later with theSchedules for Clinical Assessment in Neuropsychiatry(SCAN; Ferdinand et al., 1995), covering 61DSM-III-R diagnoses. The YASR Total Problemsscores yielded a mean r of .74 with total symptomscores on the SCAN, averaged across men andwomen. Furthermore, participants who met criteria forDSM-III-R diagnoses on the SCAN obtained significantlyhigher Total Problems, Internalizing, and Externalizingscores on the YASR than did participants whodid not meet criteria for diagnoses. In addition, theYASR Total Problems score yielded an r of -.62 withthe GAF, which is similar to the r of -.65 found in thefollow-up of American adults who had previously receivedmental health services (Achenbach et al., 1997).Thus, the associations of YASR Total Problems withdiagnostic interviews and with GAF ratings were similarin an American clinical sample and a Dutch generalpopulation sample.When reassessed 6 years later, far more of theDutch adults who obtained deviant YASR Total Problemsscores (combined borderline and clinical ranges)met criteria for interview-based DSM-IV diagnosesthan adults whose YASR scores were in the normalrange (52% vs. 7%, p

10. Validity 123Because they assess functioning in potentiallycomplementary ways, the ASEBA adult forms and theMMPI-2 may be useful to administer together in clinicaland research applications. To test associationsbetween ASEBA and MMPI-2 scales, correlationswere computed between Turkish translations of bothinstruments. The vicissitudes of translating such differentinstruments from idiomatic American English to alanguage as different as Turkish provided a strong testof whether there are significant associations betweenscales of the two instruments.The Turkish YASR and MMPI-2 were completedby 70 19- to 28-year-old students from several Turkishuniversities (unpublished data provided by Drs.Nese Erol and Isik Savasir). Table 10-9 displays rsbetween YASR and MMPI-2 scales that were significantat p

10. Validity 125functioning that apparently resulted from family counseling15 to 20 years earlier.SUMMARYThis chapter presented a variety of evidence forthe validity of ASEBA adult scores. The content validityof the problem items was supported by (a)their ability to discriminate significantly between referredand nonreferred samples, and/or (b) their significantloadings on empirically based syndromes, and/or (c) their identification by experts as being veryconsistent with DSM-IV diagnostic categories.Nearly all of the adaptive functioning items also discriminatedsignificantly between referred andnonreferred samples.Criterion-related validity of the adult scale scoreswas supported by the significant association of all butthe ABCL Intrusive syndrome scores with referral statusindependently of demographic effects. Categoricalanalyses via odds ratios and chi squares showed thatclassification of scores in the normal vs. combinedborderline and clinical ranges was significantly associatedwith referral status for nearly all ASR and ABCLscales. Discriminant analyses showed that the bestcross-validated classification rate (87% correctly classifiedas referred vs. nonreferred) was achieved bythe ASR problem items as candidate predictors. Item103. Unhappy, sad, or depressed was the prioritypredictor of referral status in analyses of both the ASRand ABCL, as it has been in many analyses of preadultsamples. As shown in Table 10-7, the probabilitythat particular Total Problems scores were fromreferred samples can be used to estimate the likelihoodthat an individual’s Total Problems score is highenough to be of concern.The complex issue of construct validity was addressedwith a variety of evidence for associations withother measures of psychopathology reported in studiesfrom Finland, the Netherlands, Turkey, and theUnited States. The evidence included significant predictiverelations from ASEBA child and adolescentsyndrome scores to the corresponding syndromes assessedwith the ASEBA adult instruments over periodsup to 10.5 years in general population and clinicalsamples; significant associations between ASEBA adultscales and diagnostic assessment; significant associationswith the Beck Depression Inventory, the BeckAnxiety Inventory, the MMPI, and the SCL-90-R;and significant associations with an intervention 15 to20 years earlier and with Child Depression Inventoryscores obtained at age 11.

Adults - Aseba

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template ?