Cognitive Interviewing Methodology

FOREWORD

As an early practitioner of cognitive interviewing, I can remember presenting many talks on this new science throughout the 1990s. Occasionally, an audience member would ask a pointed question: Although its proponents spoke of the cognitive interview as an application of psychology, were we perhaps missing something by not taking into account other disciplines as well—like linguistics, sociology, anthropology, and so on? I thought this to be a good point, despite my strong focus on cognitive psychology as an anchoring point. In fact, over the ensuing years, there have been a number of contributions that have emphasized a wider disciplinary perspective—including the argument that responses to survey questions involve more than just the individual mind of the respondent, especially as they incorporate social and cultural phenomena in a social context.

In the current volume, Kristen Miller and her colleagues provide what I believe to be the clearest statement of this truth, and the furthest point in the evolution of cognitive interviewing as a mature expression of qualitative research that provides a rich multidisciplinary perspective. The arguments, illustrations, and examples within this book challenge practitioners of cognitive interviewing—and more broadly, anyone having an interest in the subtleties of questionnaire design—to think in new ways about how survey questions are developed by designers, answered by respondents, and consumed by data users. In particular, as what I believe to be the main contribution of the volume, they expand our fundamental notion of why we choose to conduct a cognitive interview. Rather than viewing this endeavor only as an attempt to patch up deficiencies by identifying and remediating flawed survey questions, the authors conceptualize the cognitive testing enterprise as an opportunity to obtain a more comprehensive view of the items under our microscope. This interpretivist viewpoint allows us to alter our underlying research question—so that rather than asking What's wrong with the survey question?—we can conversely ask What's right with it? More to the point, we can hone that question by asking How does the question function, and what does this imply about the contexts in which it can profitably be employed? This expansive viewpoint is clearly of interest across a wide range of applications involving the use of self-report data collection instruments.

Although I use the term microscopic above, Miller et al. also further the field of cognitive interviewing by incorporating a vital macroscopic view in leading us to step back and consider the wider context of how survey items function across a range of cultures, languages, countries, and other contexts that are increasingly relevant to survey methodology. The book is the first to tackle the challenges of comparative cognitive interviewing, and takes a head-on approach to providing practical assistance to those who face the myriad challenges of question development and evaluation when faced with requirements of instrument translation, interviewing teams that speak different primary languages, and questionnaires that simply do not apply well due to cultural and structural variation. Having collaborated with Dr. Miller in particular over the recent years in which cross-cultural cognitive interviewing has taken root and grown, I can well appreciate the way she has been able to make use of battle-tested experience to save others from having to learn the same hard lessons over again.

A third unique contribution of this volume relates to analysis—well-recognized as the Achilles Heel of the cognitive interviewing process. In a word, the authors preach transparency: We need to put our cards on the table in demonstrating exactly what we mean when we say we have conducted cognitive interviews, what our data consist of, and most importantly, how we came to the conclusions we present within a cognitive testing report. Following an increasingly salient thread within the qualitative research tradition, the book provides clear examples, and conceptual direction, concerning how the results of cognitive interviews should be systematically and openly processed, so that a complete analysis is conducted. By paying significantly more attention to our analytic processes, we end up with a product that is coherent, defensible, and that sets the stage for replication and further advancement of the field as a whole.

Finally, Miller and colleagues look beyond the cognitive interview to also consider the associated pretesting approaches that exist within our ready toolbox of questionnaire development, evaluation, and testing methods. Although the notion that we can look to alternatives, such as behavior coding, psychometric, and field-based experimental studies, has deep roots in the survey methods field, the current volume advocates tying these roots together, through the use of mixed-method studies that leverage the unique strengths of each approach. In particular, the use of quantitative methods reveals how much, or how often, a phenomenon exists; whereas the overlaying of intensive qualitative methods like the cognitive interview reveals why this happens due to the richness of the information the qualitative perspective provides. In summary, the current book provides a clear pathway to new thinking, new methods, and new directions for questionnaire designers, survey managers, and data users.

GORDON WILLIS

National Cancer Institute

ACKNOWLEDGMENTS

This book has taken us somewhat longer to write than we initially anticipated. The additional time, however, brought additional critique, debate, and refinement of our ideas.

We thank Catherine Simile for providing perspective and significant insight, and Mitch Loeb for his helpful review and input. We thank our colleagues from Swan Solutions, Florencia Ramirez and Luis Cortes, for editorial comments and insurmountable help in pulling together the entire manuscript including figures, tables, bibliography, and appendices. Special thanks go to Lee Burch also of Swan Solutions for his many years of inspiration and support, as well as Karen Whitaker—office manager extraordinaire—who continuously reminds us to think about the big picture while keeping us on task in the here and now. We are especially grateful for all our colleagues in the Questionnaire Design Research Laboratory at the National Center for Health Statistics (NCHS) who, collectively, have helped to improve cognitive interviewing methodology.

We also thank the members of the question evaluation community who developed and sharpened the field over the past 20 years. We are particularly grateful for conversation (and sometimes loud debate!) with Gordon Willis, Norman Bradburn, Janet Harkness, Jack Fowler, Paul Beatty, Fred Conrad, Terry DeMaio, Jennifer Rothgeb, Peter Mohler, Rory Fitzgerald, and Debbie Collins—all of whom helped to shape our thinking.

Additionally, we thank our institutions: the National Center for Health Statistics along with the NCHS Office of Research and Methodology which, under the direction of Nat Schenker, promoted and prioritized question evaluation methodology, providing us the resources and time to develop this work. The University of Granada and the Spanish National Statistics Institute, particularly, Miguel Angel Martínez Vidal who pushed the cognitive interviewing projects in Spain.

We are appreciative of Wiley and our editors, Sari Friedman and Steve Quigley, for realizing the value of this project.

A most special thank you to the NCHS Associate Director of Science, Jennifer Madans, who for over a decade pushed us, argued with us, forced us to articulate better (and sometimes drove us crazy!) more than anyone else. Without her mentorship and sincere dedication to question evaluation and the advancement of survey methodology, this book would not exist. For this, we are truly grateful.

CONTRIBUTORS

ISABEL BENITEZ BAENA, University of Granada, Spain

VALERIE CHEPP, Hamline University

CAROLINE GRAY, Research Institute of the Palo Alto Medical Foundation

MEREDITH MASSEY, National Center for Health Statistics

JUSTIN MEZETIN, National Center for Health Statistics

KRISTEN MILLER, National Center for Health Statistics

JOSÉ-LUIS PADILLA, University of Granada, Spain

J. MICHAEL RYAN, The American University in Cairo

PAUL SCANLON, National Center for Health Statistics

ALISÚ SCHOUA-GLUSBERG, Research Support Services

ANA VILLAR, City University London

GORDON WILLIS, National Cancer Institute

STEPHANIE WILLSON, National Center for Health Statistics

1

Introduction

KRISTEN MILLER

National Center for Health Statistics

Although the beginnings of survey research can be traced as far back as the late 1880s, the discussion of question design and the need for standardized questions did not appear for another 50 years (Groves et al. 2009). Since this time, notions about question design have dramatically transformed, particularly in regard to question evaluation. In 1951, Stanley Payne published his book, The Art of Asking Questions, and laid out 100 considerations for writing survey questions. Although he maintained that question evaluation studies could be helpful, he argued that the actual writing process should be the higher concern. Today, however, there is a greater emphasis on question evaluation. Also, with the entrance of psychologists, psychometricians, and more recently, anthropologists, qualitative methodologists, and cognitive sociologists, the scientific rigor and scope have increased.

A significant advancement for question evaluation occurred in the 1980s with the entrance of cognitive psychology and the study of the cognitive aspects of survey methodology (CASM). The CASM movement not only brought attention to the issue of measurement error, it also established the idea that individual processes, specifically, respondents' thought processes, must be understood to assess the validity and potential sources of error (Schwarz 2007). The underlying supposition is that, as noted by Willis (2005), the respondent's cognitive processes drive the survey response, and an understanding of cognition is central to designing questions and to understanding and reducing sources of response error (p. 23). Thus, with the advent of CASM, the focus of question design shifted from the question writer to the respondent and cognitive processes.

The cognitive processes that make up question response have been represented in a number of theoretical models. A commonly cited question-response model contains four stages: comprehension, retrieval, judgment, and response (Tourangeau 1984; Tourangeau et al. 2000; also see Willis 2005 for a detailed discussion). To provide a response, each respondent proceeds through four specific steps: (1) determining what the question is asking, (2) recalling or retrieving the relevant information, (3) processing the information to formulate an answer, and (4) mapping that answer onto the provided response categories. By recognizing the cognitive processes, it is possible to understand the complexity of the question-response process as well as the numerous possibilities for response error (Tourangeau et al. 2000; Willis 2004, 2005). By establishing a theoretical foundation for survey question response, the CASM movement provided a basis for scientific inquiry as well as a practical basis for understanding and reducing response error in survey data.

Today there is little debate that question design—how questions are worded and the placement of those questions within the questionnaire—impacts responses (e.g., Fowler 2009; Krosnick and Presser 2010). Newly developed or re-conceptualized methodologies (e.g., latent class analysis, behavior coding, and item-response theory) have repeatedly demonstrated the impact of question design (Madans et al. 2011). Psychometricians, for example, have shown that scale items with more response categories are increasingly likely to produce response distributions with a wider spread than those with fewer categories (Crocker and Algina 2008). Split sample experiments—a method that divides a survey sample into two groups whereupon one group receives a question and the other receives a different version of the same question—have also shown varying estimates (Krosnick 2011; Fowler 2004). In terms of substance and practicality, each methodology has its own benefits but also limitations (see Madans et al. 2011 for in-depth explication). The future of question evaluation lies in the use and integration of varying methodologies. Understanding the range of methodological perspectives—the suppositions, benefits, and limitations—will improve knowledge of question response and survey error, and ultimately ensure quality survey data.

1.1 COGNITIVE INTERVIEWING METHODOLOGY