Educational Design Research

11
Susan McKenney and Thomas C. Reeves

Abstract
Educational design research is a genre of research in which the iterative development of
solutions to practical and complex educational problems provides the setting for scientific
inquiry. The solutions can be educational products, processes, programs, or policies.
Educational design research not only targets solving significant problems facing educa-
tional practitioners but at the same time seeks to discover new knowledge that can inform
the work of others facing similar problems. Working systematically and simultaneously
toward these dual goals is perhaps the most defining feature of educational design research.
This chapter seeks to clarify the nature of educational design research by distinguishing it
from other types of inquiry conducted in the field of educational communications and tech-
nology. Examples of design research conducted by different researchers working in the field
of educational communications and technology are described. The chapter concludes with
a discussion of several important issues facing educational design researchers as they pursue
future work using this innovative research approach.

Keywords
Design research • Design-based research • Formative research • Design experiments

Introduction can be educational products (e.g., a multiuser virtual world

learning game), processes (e.g., a strategy for scaffolding
Educational design research is a genre of research in which student learning in online courses), programs (e.g., a series
the iterative development of solutions to complex educa- of workshops intended to help teachers develop more effec-
tional problems provides the setting for scientific inquiry. tive questioning strategies), or policies (e.g., year-round
The solutions that result from educational design research schooling). Educational design researchers attempt to solve
significant real world problems while at the same time they
seek to discover new knowledge that can inform the work of
others facing similar problems. This chapter summarizes
arguments and evidence presented by Barab and Squire
S. McKenney (*) (2004), Burkhardt (2009), Reeves (2011), Schoenfeld (2009),
Open University of the Netherlands & University of Twente, OWK/ van den Akker, Gravemeijer, McKenney, and Nieveen
GW/UTwente, PO Box 217, Enschede 7500 AE, The Netherlands
e-mail: [email protected]
(2006a), and others that educational design research is an
innovative and exceptionally promising approach to improv-
T.C. Reeves
College of Education, The University of Georgia,
ing the quality and impact of educational research in general,
325C Aderhold Hall, Athens, GA 30602-7144, USA and educational communications and technology research in
e-mail: [email protected] particular.

J.M. Spector et al. (eds.), Handbook of Research on Educational Communications and Technology, 131
DOI 10.1007/978-1-4614-3185-5_11, © Springer Science+Business Media New York 2014
132 S. McKenney and T.C. Reeves

limitations of laboratory settings, the value of relinquishing

Educational Design Research Origins control of variables in return for increased ecological validity
of the findings has been gaining support over the last 30
Design research is not defined by its methods but by the goals of years. In 1992, two landmark papers were published which
those who pursue it. Design research is constituted within com-
are often credited with launching educational design research
munities of practice that have certain characteristics of innova-
tiveness, responsiveness to evidence, connectivity to basic science, as a specific genre of scientific inquiry. Brown’s (1992) article
and dedication to continual improvement. Bereiter (2002) p. 321. in the Journal of the Learning Sciences discussed tensions
between laboratory studies of educational innovations and
What has prompted scholars around the globe sharing the challenges inherent in integrating these innovations into real
above-mentioned characteristics of “innovativeness, respon- world classrooms as background to describing her own
siveness to evidence, connectivity to basic science, and dedi- design experiments. That same year, Collins (1992) pub-
cation to continual improvement” to come together in the lished a book chapter arguing that education should be
pursuit of educational design research? At least two main viewed as a design science akin to aeronautics, as opposed
motives can be identified. Interestingly, both perspectives on an analytical science similar to physics, emphasizing the
have strong historical ties to educational psychology, and fact that laboratory conditions could rarely approximate
both perspectives are concerned with making a contribution conditions in real classrooms.
to educational practice. The first motive is driven more by By the turn of the millennium, support was increasing for
what society needs while the second has more to do with innovative research approaches that might yield the kind of
finding adequate methods to meet those needs. knowledge that can be put to use for the improvement of edu-
First, stemming from the notion that scientific understand- cation. Advocates for these new approaches accepted that the
ing should be used to solve or at least gain a better under- kinds of knowledge needed would have to be constructed in
standing of practical problems, the call for scientific inquiry the complex “laboratories” of everyday learning environments
to yield what Lagemann (2002) refers to as “usable knowl- such as classrooms or online courses. The establishment of
edge” has been present for over a century. Although this educational design research is growing steadily. This momen-
focus on demonstrable impact may be ignored by some who tum became apparent through several special issues of highly
recommend that educational researchers should emulate the respected journals, including Educational Researcher (2003,
methods of the so-called hard sciences (e.g., physics) that 31(1)), Journal of the Learning Sciences (2004, 13(1)), and
seek knowledge without expectation of practical application, Educational Psychologist (2004, 39(4)). Since then, several
the expectation for social science research to connect funda- books have been written about educational design research.
mental understanding with applied use dates back to Books have focused on conceptualization (van den Akker
Münsterberg (1899) and Dewey (1900), if not earlier. Both et al., 2006a) methodological considerations (Kelly, Lesh, &
of these former American Psychological Association presi- Baek, 2008), and the details of conducting design studies
dents expressed the need for a linking science, which would (McKenney & Reeves, 2012) across educational fields.
use empirical insights and theoretical advancements to Related volumes have appeared specifically in the domains
inform problem-solving and improvement initiatives in prac- of literacy (Reinking & Bradley, 2008) and instructional
tice. This call has been taken up gradually within the fields of design (Richey & Klein, 2007). In addition to special issues
education and psychology, for example in the work of Robert and books about educational design research, numerous
Glaser (1976) who laid out the elements of a psychology of reports of educational design research initiatives have been
instruction and called for a science of design in education. published in research journals such as Instructional Science
Donald Stokes (1997), an American political scientist, pro- (cf. Xie & Sharma, 2011), the Journal of the Learning
vided a fresh look at the goals of science and their relation to Sciences (e.g., Schwarz & Asterhan, 2011), the Journal of
application to real world problems, in his highly acclaimed Research on Technology in Education (e.g., Basham, Meyer,
book titled, Pasteur’s Quadrant: Basic Science and & Perry, 2010), and Educational Technology Research and
Technological Innovation. Stokes promoted more “use- Development (e.g., Reynolds & Caperton, 2011). In addition,
inspired basic research” akin to the work of the French chem- doctoral dissertations using educational design research have
ist and microbiologist, Louis Pasteur. He contrasted Pasteur’s been completed at multiple institutions such as the University
pragmatic research approach with that of the basic science of California, Berkeley (e.g., Brar, 2010), Unversity of
goals of Danish physicist, Niels Bohr, and the applied Florida (e.g., Drexler, 2010), the University of Georgia
research aims of the American inventor, Thomas A. Edison. (e.g., Oh, 2011), the Pennsylvania State University (e.g., Lee,
Second, educational researchers have been searching for 2009), and the University of Twente (e.g., Raval, 2010).
adequate methods to yield the kinds of empirical insights Today we see many sectors within education that seem to
and theoretical advancements that could be used to address embrace educational design research, including: learning
real concerns in educational practice. Acknowledging the sciences, instructional design, curriculum development and
11 Educational Design Research 133

teacher professional development. While educational design research in which the iterative development of solutions
research is not inherently tied to any specific subject area, (e.g., educational products, processes, programs or policies)
much of the work published so far has been related to science to practical and complex educational problems, provides the
or mathematics, perhaps because more funding has been setting for scientific inquiry, and yields new knowledge that
available for research related to STEM (science, technology, can inform the work of others. Working systematically and
engineering, and mathematics) disciplines than for other simultaneously toward these dual goals may be considered
areas (Kelly et al., 2008). However, educational design the most defining feature of educational design research.
research is also being increasingly used in language and lit- Educational design research is not a methodology. It uses
eracy research (Reinking & Bradley, 2008), as well as other quantitative, qualitative and—probably most often—mixed
disciplines. A wide variety is present across educational methods to answer research questions. In so doing, educa-
design study literature, a development that is partly accounted tional design research is held to the same standards as other
for by the methodological traditions within the various scientific work when it comes to providing transparency of
educational sectors, individual researcher preferences and the process and adequate warrants for the knowledge claims
the resources available for specific projects. In addition, vari- it yields (cf. Shavelson, Phillips, Towne, & Feuer, 2003).
ance across the twofold motives driving educational design In addition to the knowledge generated, the value of educa-
research plays a large role in explaining the diversity of these tional design research is measured in terms of its ability to
kinds of studies. While pursuing both goals simultaneously improve educational practice (Design-Based Research
remains a defining feature of educational design research, Collective, 2003).
one goal may feature more prominently than the other. For
example, relating more to the motive of improving practice,
educational design research may be conducted primarily to: How Does Educational Design Research
• Solve a problem (e.g., increase the participation of women Compare to Other Approaches?
and other minorities in engineering and science careers),
• Put knowledge to innovative use (e.g., use the affordances While both are concerned with developing new knowledge
of smart phones to enable mobile learning), and/or and are connected to design processes, educational design
• Increase robustness and systematic nature of design practices research has commonalities but also differences from the
(e.g., establish a set of design principles for implementing instructional design focused design and development
inquiry-based learning in middle school science). research described by Richey and Klein (2007, and in this
Or, relating more to the motive of enhancing the quality of volume). If considered as a Venn diagram, educational design
research findings, educational design research may be con- research and design and development research would overlap
ducted primarily to: in projects that are concerned with actively solving problems
• Generate new knowledge (e.g., develop a theory of game- in educational practice (e.g., design and testing of software
based learning), to help plan lessons). The area that would be unique to design
• Generate different types of knowledge (e.g., enhance and and development research would be those projects that are
extend knowledge related to professional development for concerned with developing tools or models to support educa-
scaffolding strategies for math teachers), and/or tion in the long run, but that do not function as educational
• Increase the ecological validity of research-based knowledge interventions (e.g., retrospective analysis of how instruc-
(e.g., increase the likelihood that educational innovations tional designers carry out their tasks). Design research projects
will be used to transform educational practice). that would not overlap with design and development research
would be those not specifically concerned with advancing
the field of instructional design (e.g., design and testing of a
Clarifying the Nature of Educational Design learning sequence for early literacy).
Research Educational design research is also different from evalua-
tion research (Clarke, 1999), although formative and summa-
What Is Educational Design Research? tive evaluation methods are among the main vehicles used to
study and fine-tune interventions in both cases. First, problem
While studies do differ in terms of which motives are more definition and solution design are rarely featured in evalua-
powerful determinants in shaping the inquiry, educational tion research. Second, a key difference is that evaluation
design research in general distinguishes itself from other research is primarily concerned with evaluating and possibly
forms of inquiry by attending to both solving problems by improving the qualities of a particular intervention. The
putting knowledge to use, and through that process, generat- broader scientific orientation of generating usable knowledge
ing new knowledge. As stated elsewhere (McKenney & (e.g., in the form of models to underpin design, theories about
Reeves, 2012), educational design research is a genre of how teachers learn, descriptions of what engages learners, etc.)
134 S. McKenney and T.C. Reeves

is not as overtly present in evaluation research as in educa- change in a particular educational context. Educational
tional design research. design research is iterative because it generally evolves
Educational design research also entails more than through multiple cycles of design, development, testing, and
research-based educational design. They are both forms of revision. It is collaborative because it requires the expertise
scientific inquiry, and often, each values a rational approach. of multidisciplinary partnerships, including researchers and
They both embrace systems thinking and are both shaped by practitioners, but also often others (e.g., subject matter spe-
iterative, data-driven processes to reach successive approxi- cialists, software programmers or facilitators). Educational
mations of a desired intervention. However, research-based design research is adaptive because the intervention design
educational design focuses solely on intervention develop- and sometimes also the research design are often modified in
ment, whereas design research strives explicitly to make a accordance with emerging insights. Finally, it is theory-
scientific contribution—of value to others outside the oriented not only because it uses theory to ground design,
research/design setting—in addition to the intervention but also because the design and development work is under-
development. This has important implications for the entire taken to contribute to a broader scientific understanding.
process. Additional information on these differences is avail-
able in (McKenney & Reeves, 2012; Oh & Reeves, 2010).
Similarly, action research (cf. Mills, 2002) also lacks the Process
emphasis on finding the kind of robust public knowledge that is
a hallmark of educational design research. There is no set process for conducting the “manifold enter-
Distinguishing educational design research from other prise” (Bell, 2004, p. 245) of educational design research.
forms of inquiry in education is made more difficult because This approach to inquiry is rich with variation in terms of
it has been referenced in the literature by a number of differ- models and frameworks that describe, and in a few cases,
ent terms such as “design-based research” (cf. Barab & guide the process. Across that variation, some similarities
Squire, 2004), “design experiments” (cf. Brown, 1992), can be identified:
“development research” (cf. van den Akker, 1999), “formative • Educational design research uses scientific knowledge
experiments” (cf. Reinking & Bradley, 2008), “formative (and to varying degrees, also other kinds of knowledge
research” (cf. Newman, 1990), and simply “design research” such as craft wisdom) to ground design work
(cf. Kelly et al., 2008). There are subtle differences in how • Educational design research produces scientific knowl-
these terms are used by various researchers as delineated edge (and in some cases, also craft wisdom among the
in McKenney and Reeves (2012). The term “educational participants)
design research” is used in this chapter and elsewhere (cf. • Though the terminology and contents differ, three phases
Plomp & Nieveen, 2009; van den Akker, Gravemeijer, can be distinguished in educational design research: an
McKenney, & Nieveen, 2006b) because including the word analysis/orientation phase; a design/development phase;
“educational” in the term helps to avoid confusion with and an evaluation/retrospective phase; these are often
design research as used in other fields. For example, Laurel’s revisited in the lifespan of a project
(2003) book simply titled Design Research concerns the • Educational design research strives to develop both inter-
field of human computer interface design and industrial engi- ventions in practice and reusable knowledge
neering rather than education.

Rich Variation
Conducting Educational Design Research
Thought-provoking differences in design research are also
Characteristics present. Some of the differences stem from the units of anal-
ysis, scope of implementation, nature of the subject areas
Characteristics of educational design research have been addressed, or from the research domains and methodological
offered in the literature (Kelly, 2003; Reinking & Bradley, traditions in which studies originate. As mentioned earlier,
2008; van den Akker et al., 2006a; Wang & Hannafin, 2005). the relative emphasis on each motive (solution development,
Common descriptors include: pragmatic, grounded, inter- new knowledge or equally on both) can also wield strong
ventionist, iterative, collaborative, adaptive and theory- influence on the design research process. But other differ-
oriented. Educational design research is pragmatic because it ences stem from the concerns of those interpreting the
is concerned with generating usable knowledge, and usable concept and conducting the studies.
solutions to problems in practice. It is grounded because it McKenney and Reeves (2012) surveyed models for edu-
uses theory, empirical findings and craft wisdom to guide the cational design research and, in addition to highlighting
work. It is interventionist because it is undertaken to make a similarities like those mentioned above, noted unique con-
11 Educational Design Research 135

Implementation & Spread

Maturing
Analysis Design Evaluation Intervention

Theoretical
Exploration Construction Reflection
Understanding

Fig. 11.1 Generic model for conducting educational design research (McKenney & Reeves, 2012)

tributions each one has to offer. The Osmotic Model, offered (2012) created a generic model for design research (see
by Ejersbo et al. (2008), depicts the parallels of the design Fig. 11.1). Through this basic visualization, this model shows
cycle and the research cycle. The authors point out that both only the core elements of a flexible process that features
cycles originate from the problem and would ideally run the three main stages described earlier, taking place in inter-
simultaneously, but state that this ideal is often not the case. action with practice and yielding the dual outputs of knowl-
Bannan-Ritland and Baek (2008) developed the Integrated edge and intervention.
Learning Design Framework, which depicts four main stages
and across those, 14 steps, in a combined approach to
research and development. Along with the process model, Scientific Outputs
guiding questions for research and examples of applicable
methods for each main phase are given. Reeves (2006) pre- Different terms have been used to describe the kinds of theo-
sented a minimalist model that highlights four main phases retical knowledge that are produced by educational design
of design research: problem analysis; solution development; research (cf. Edelson, 2002; McKenney & Reeves, 2012; van
iterative refinement; and reflection to produce design princi- Aken, 2004; van den Akker, 1999). Descriptive, substantive
ples. He compared these phases to the four phases of predic- or declarative knowledge is generated to describe certain
tive research. In contrast to the aforementioned three models, phenomena (e.g., what learner behaviors are triggered by
McKenney, van den Akker, and Nieveen (2006) offered a certain prompts). Prescriptive or procedural knowledge is
model which is more conceptually oriented than process- generated to help inform interventions in practice (e.g., how
oriented. This model depicts tenets guiding a research and to facilitate learning through the strategic use of certain
development cycle, situated in a particular context, yielding prompt types under certain circumstances). Some projects
three main outcomes: professional development of the par- may develop a research agenda more attuned to one type of
ticipants; the designed intervention; and design principles. knowledge over another, though eventually attending to both
In addition to these visual models, Gravemeijer and Cobb types seems to be the case more often than not.
(2006) described important steps in the three main phases of Different terms have been used in literature to describe the
their work: preparing for a design experiment; conducting a kind of integrated procedural and declarative knowledge that
design experiment; and retrospective analysis. Based on a comes out of design research, but design principles is proba-
review of literature, Wang and Hannafin (2005) delineated bly the most prevalent (cf. Kali, 2008; Kim & Hannafin, 2008;
and argued for nine principles of design-based research. Mishra & Koehler, 2006; Quintana, Reiser, Davis, Krajcik,
Finally, Reinking and Bradley (2008) posed six questions as Fretz, Duncan et al., 2004; van den Akker, 1999). Bell,
a guide for conducting formative experiments, relating to: Hoadley, and Linn (2004) describe design-principles as:
pedagogical goals; classroom intervention; factors affecting …an intermediate step between scientific findings, which must
the intervention; modifications to the intervention; unpre- be generalized and replicable, and local experiences or exam-
dicted effects of the intervention; and changes in the instruc- ples that come up in practice. Because of the need to interpret
tional environment due to the intervention. design-principles, they are not as readily falsifiable as scientific
laws. The principles are generated inductively from prior exam-
Based on their survey and analysis of existing models ples of success and are subject to refinement over time as others
and frameworks for design research, McKenney and Reeves try to adapt them to their own experiences. (p. 83).
136 S. McKenney and T.C. Reeves

Table 11.1 Three examples demonstrating educational design research variation

Thomas et al. (2009) Klopfer and Squire (2008) Oh (2011)
Problem Middle school students were High school and college students were frequent Graduate student collaboration in online
relatively unengaged in users of handheld devices such as smart phones, learning course was superficial and
meaningful scientific inquiry but were not using them to learn unproductive
Main focus Investigating the implementation Developing innovative applications for mobile To optimize collaborative group work and
of a technology-rich educational computing for environmental science education student learning in an online higher
innovation in a public elementary education learning environment
school in the USA
Intervention Quest Atlantis: a 3D multiplayer A series of games that can be played on handheld “E-learning Evaluation” course based on
developed virtual environment devices such as PDA and smart phones authentic tasks for online delivery
Knowledge Theory of transformational play Theoretical framework called “augmented Multiple design principles and associated
created reality educational gaming” strategies to enhance group work in online
courses
Research Observations Observations Participant observations
methods Interviews Interviews Questionnaires
used Surveys Focus groups Interviews
Document analyses Discourse analysis Three sequential case studies
Three qualitative case studies Case studies
Design narratives
Research This design research initiative The design research study has been underway This study lasted 2 years with no direct
scope has been underway for more than since 2001 with initial funding from Microsoft funding
a decade with substantial funding and other sources
from NSF and other sources
Primary As of 2010, Quest Atlantis had The work started with this project is now part of the An online course design for a graduate level
practical been used by 50,000 students in Games, Learning, and Society group at the course based around authentic tasks was
contribution more than a dozen countries. University of Wisconsin where numerous learning developed with substantial support for group
atlantis.crlt.indiana.edu games can be found. www.gameslearningsociety.org work. http://authenticlearning.info/
AuthenticLearning/Home.html

On the other hand, van den Akker (1999) suggests that the study either the intervention itself (e.g., as a type of interven-
knowledge encompassed in design principles can be con- tion for which guidelines or design frameworks are needed);
veyed through heuristic statements, such as, “If you want to or phenomena that are engendered by the interventions (e.g.,
design intervention X [for purpose/function Y in context Z]; learner reactions).
then you are best advised to give that intervention the
characteristics C1, C2, …, Cm [substantive emphasis]; and
do that via procedures P1, P2, …, Pn [procedural emphasis]; Examples
because of theoretical arguments T1, T2, …, Tp; and empiri-
cal arguments E1, E2, … Eq.” (p. 9). Complementing these Different research reports are used here (Klopfer & Squire,
perspectives on design principles, Linn and Eylon (2006) 2008; Oh, 2011; Thomas, Barab, & Tuzun, 2009) to illustrate
also describe design patterns, which illustrate promising the variety of educational design research conducted within
instructional sequences, and may be guided or fine-tuned by the field of educational communications and technology.
design principles. One study (Thomas et al., 2009) was conducted by a research
team led by Sasha Barab, one of the most highly respected
senior professors in the field, with substantial funding from
Practical Outputs the National Science Foundation and other sources; one
study was co-led by an at-the-time early career assistant pro-
In educational design research, research and development fessor, Kurt Squire, with start-up funding from Microsoft
are integrated to create educational interventions that address and other sources; and the last was carried out by a doctoral
practical problems. In early stages, this involves analysis of student, Eunjung Oh, working with one other doctoral stu-
the problem to be addressed. Using the findings from a needs dent and a practitioner with no funding beyond a graduate
and context analysis, together with a clarified problem state- teaching assistantship. For each one, the problem addressed,
ment, design work commences. Depending on the scope of the primary focus of the research, the intervention that was
the project, (re-)design work can last from several weeks to developed, the theoretical contributions, the methods used,
several years. Especially the revisions are fed by field the scope of the intervention involved as well as its practical
investigations using a range of strategies and methods to contribution are summarized in Table 11.1.
11 Educational Design Research 137

The three examples described here illustrate how different Information Richness and Efficiency: Seeking
types of research reports are published as sub-components of a Productive Balance
larger educational design research projects. Published in the
Journal of Educational Computing Research, Thomas et al. When conducting educational design research, it is necessary
(2009) is one of a series of journal papers in which Barab and to address questions about appropriate tactics for increasing
his colleagues have described their efforts to refine a theory the information richness and efficiency of data collection
of transformational play while at the same time seeking to procedures and instruments without being over-whelmed
develop advanced forms of interactive learning games. This with data. Design researchers should not be driven by the
paper summarizes the results of three qualitative studies misconception that “more is better.” This notion is aptly
focused on the challenges and successes involved in imple- conveyed by Dede (2004, p. 107) who noted in reference to
menting Quest Atlantis, a 3D multiplayer virtual environ- a design study that “everything that moved within a 15-foot
ment (MUVE), which serves as the primary vehicle for radius of the phenomenon was repeatedly interviewed, video-
instantiating Barab’s transformational play learning theory taped, surveyed and so-forth—this elephantine effort resulted
and for allowing it to be refined through iterative design- in the birth of mouse-like insights in their contribution to
based research. educational knowledge.”
Published in the Educational Technology Research and
Development Journal, Klopfer and Squire (2008) describe a
multi-year project to enhance student learning related to Optimizing Processes: Stacking Smaller
environmental science through the development and Studies Together
refinement of learning games that are accessed with hand-
held devices such as PDAs and smart phones. In addition to Other questions arise around the linkages among design,
developing an array of learning games, the project has sought prototyping, implementation, data collection, processing,
to develop and refine a theoretical framework called “aug- analysis, and re-design. Managing the process of communi-
mented reality educational gaming” that can be applied by cating evaluation findings and subsequently utilizing them
other games designers. The paper provides considerable for improvement of interventions is difficult. Realistic time-
detail about the development of the learning games using a lines must be established with allowances for flexibility.
unique “design narrative” approach. This particular paper Educational design research projects must inevitably be
focuses on iterative design cycles based on five case studies divided into smaller, more manageable chunks. These chunks
conducted in real high school classrooms. and the smaller studies involved in them can function as
Oh (2011) reports the findings of a doctoral dissertation “bricks” in a larger structure that forms both the evolving
that pursued two primary goals: (1) optimizing collaborative intervention and the refined knowledge. Emerging insights
group work in an online graduate level course focused on can be shared through shorter (e.g., article-sized) reports of
“E-Learning Evaluation,” and (2) developing a refined model smaller chunks, whereas books or other media might be more
of group work in online courses and identifying design prin- appropriate for sharing new knowledge derived from the
ciples for supporting online collaborative group work among whole of long-term efforts. Often, the interim (i.e., smaller
adult learners The dissertation provides a comprehensive chunk) reporting stands on its own and does not (need to)
portrayal of a 2-year design research project using what mention the larger study; also, interim reporting for an exter-
Boote and Beile (2005) called the “compilation of research nal audience can be a timely vehicle for fostering reflection
articles” (p. 10) format for dissertations. The dissertation among design research team members.
includes one published article, three submitted papers, one
detailed methodology chapter, and one detailed results
chapter. Oh (2011) documents how mixed methods were Measuring Impact: Powerful Examples Needed
applied across several semester-length iterations of an online
course to yield multiple distinct design principles for sup- Ultimately, educational design researchers must address ques-
porting group work by adults. tions regarding the most relevant indicators of quality, suc-
cess and impact of the interventions and knowledge advances
that result from their efforts. Burkhardt (2006) writes about
Addressing Inherent Challenges what is needed to bring about greater acceptance of educa-
tional design research. He describes several Nobel Prize win-
Inspired by van den Akker’s (1999) design research chal- ners for design and development in other fields and concludes
lenges, this section briefly touches on several important that educational design research candidates should be assessed
issues that often crop up in educational design research, how on the basis of their: impact on practice; contribution to the-
they may be attended to, and areas that require further ory and/or knowledge; and improvement in either research
consideration. and/or design methodology. While it is surely too early to be
138 S. McKenney and T.C. Reeves

expecting Nobel Prizes for educational design researchers, engineering and product design tend to embrace creativity
this approach will only gain wide acceptance when it can be more than most educational researchers (e.g., Laurel, 2003).
shown to make the much-needed gains in demonstrating the Another perspective can be found in appreciative inquiry in
impact educational research (cf. Kaestle, 1993). health care (e.g., Carter et al., 2007) that emphasizes design
based on opportunity, as opposed to patching gaps uncovered
by reductionist problem diagnostics.
Generalizability: Toward Uptake and Use Since the landmark design research articles in 1992, a
of New Knowledge growing appreciation for educational design research in a
wide variety of contexts has been evident (Anderson &
The main conceptual vehicle through which new knowledge Shattuck, 2012). Gradually, the design research literature is
is transferred outside of the research context, generalizability beginning to show more consideration of factors that affect
means different things to different researchers. All research- implementation. Instead of tossing innovations over the met-
ers must seek to identify promising approaches to enable aphorical walls of classrooms and online learning environ-
uptake and use of research findings. Because educational ments, educational design researchers are working hand in
design research takes place in natural settings where more hand with practitioners to conduct design and research in ways
variables are present than can be controlled for, the findings that make substantive change possible. The importance of col-
from these studies cannot yield immutable rules, easily trans- laborative approaches and on-the-ground understanding of
ferred without consideration. But they can yield useful implementation issues, which were privileged topics of
insights to inform the work of others (design work or other- research in the 1970s (cf. Fullan & Pomfret, 1977; Hall,
wise). For example, when designs are tested in multiple set- Wallace, & Dossett, 1973; Havelock, 1971) seem relatively
tings and under varying conditions, or when design features are new—but also quite dear—to many of those currently practic-
systematically varied under similar conditions, theory develop- ing design research. Some researchers emphasize this perspec-
ment can occur through analytic generalization. According to tive by referring to their work as design-based implementation
Yin (1989, p. 44), analytic generalization is a process through research (e.g., Penuel, Fishman, Cheng, & Sabelli, 2011).
which “the investigator is striving to generalize a particular set We embrace the surge of interest in these concerns, and
of results to a broader theory” which can be of use to others. express our hope for a renaissance of scholarship that brings
Alternatively, knowledge produced through design research researcher and practitioner expertise together to bear on sub-
can be shared and used through case-to-case generalization. stantial educational issues (McKenney & Reeves, 2013).
Firestone (1993) refers to case-to-case generalization as the Educational design research is one of several genres of
transfer of ideas that takes place when a person in one setting inquiry that can lead the way in contributing to scientific
considers adopting an intervention, or its underlying proposi- understanding in the long term through its study of meaningful
tions and frameworks in another setting. To do this, the knowl- implementation in the here and now.
edge producer is obligated to explicate how the specific instance
studied compares to other instantiations of the phenomenon.
In so doing, description of salient characteristics of both the References
intervention and the context in which it is enacted are essential.
Clearly, when it comes to putting the knowledge of design Anderson, T., & Shattuck, J. (2012). Design-based research: A decade
research to use, the knowledge producer must portray the work of progress in education research? Educational Researcher, 41(1),
16–25.
well enough. This could mean, for example, adhering to Lincoln
Bannan-Ritland, B., & Baek, J. (2008). Teacher design research: An
and Guba’s (1985) criteria for naturalistic inquiry: credibility, emerging paradigm for teachers’ professional development. In A.
transferability, dependability and confirmability (parallel to Kelly, R. Lesh, & J. Baek (Eds.), Handbook of design research
internal validity, external validity, reliability and objectivity, methods in education: Innovations in science, technology, engineering,
and mathematics learning and teaching. London: Routledge.
respectively). At the same time, knowledge consumers are
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake
obliged to critically assess the applicability of certain ideas in the ground. The Journal of the Learning Sciences, 13(1), 1–14.
for their own specific contexts. Basham, J. D., Meyer, H., & Perry, E. (2010). The design and applica-
tion of the digital backpack. Journal of Research on Technology in
Education, 42(4), 339–359.
Bell, P. (2004). On the theoretical breadth of design-based research in
On the Horizon education. Educational Psychologist, 39(4), 243–253.
Bell, P., Hoadley, C., & Linn, M. (2004). Design-based research in edu-
Educational design researchers and arguably all educational cation. In M. Linn, E. Davis, & P. Bell (Eds.), Internet environments
of science education (pp. 73–85). Mahwah, NJ: Lawrence Earlbaum
researchers must seek to balance rigor and relevance (Reeves,
Associates.
2011). To find this balance, educational design researchers Bereiter, C. (2002). Design research for sustained innovation. Cognitive
might do well to learn from sister fields. For example, Studies, 9(3), 321–327.
11 Educational Design Research 139

Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the Kaestle, C. F. (1993). The awful reputation of education research.
centrality of the dissertation literature review in research preparation. Educational Researcher, 22(1), 23, 26–31.
Educational Researcher, 34(6), 3–15. Kali, Y. (2008). The design principles database as means for promoting
Brar, R. (2010). The design and study of a learning environment to design-based research. In A. Kelly, R. Lesh, & J. Baek (Eds.),
support growth and change in students’ knowledge of fraction Handbook of design research methods in education (pp. 423–438).
multiplication. Unpublished doctoral dissertation, The University of London: Routledge.
California, Berkeley. Kelly, A. (2003). Research as design. Educational Researcher, 32(1),
*Brown, A. L. (1992). Design experiments: Theoretical and method- 3–4.
ological challenges in creating complex interventions in classroom *Kelly, A., Lesh, R., & Baek, J. (Eds.). (2008). Handbook of design
settings. Journal of the Learning Sciences, 2(2), 141–178. research methods in education. New York, NY: Routledge.
Burkhardt, H. (2006). From design research to large-scale impact: Kim, H., & Hannafin, M. (2008). Grounded design of web-enhanced
Engineering research in education. In J. van den Akker, K. case-based activity. Educational Technology Research and
Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational Development, 56, 161–179.
design research. London: Routledge. Klopfer, E., & Squire, K. (2008). Environmental detectives—The
Burkhardt, H. (2009). On strategic design. Educational Designer: development of an augmented reality platform for environmental
Journal of the International Society for Design and Development in simulations. Educational Technology Research and Development,
Education, 1(3). Available online at: http://www.educationalde- 56(2), 203–228.
signer.org/ed/volume1/issue3/article9/index.htm Lagemann, E. (2002). An elusive science: The troubling history of
Carter, C. A., Ruhe, M. C., Weyer, S. M., Litaker, D., Fry, R. E., & education research. Chicago: University of Chicago Press.
Stange, K. C. (2007). An appreciative inquiry approach to practice Laurel, B. (2003). Design research: Methods and perspectives.
improvement and transformative change in health care settings. Cambridge, MA: MIT Press.
Quality Management in Health Care, 16(3), 194–204. Lee, J. J. (2009). Understanding how identity supportive games can
Clarke, A. (1999). Evaluation research: An introduction to principles, impact ethnic minority possible selves and learning: A design-based
methods and practice. London: Sage. research study. Unpublished dissertation study, The Pennsylvania
*Collins, A. (1992). Toward a design science of education. In E. State University.
Lagemann & L. Shulman (Eds.), Issues in education research: *Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury
problems and possibilities (pp. 15–22). San Francisco, CA: Jossey- Park, CA: Sage Publications.
Bass. Linn, M., & Eylon, B. (2006). Science education: Integrating views of
Dede, C. (2004). If design-based research is the answer, what is the learning and instruction. In P. Alexander & P. Winne (Eds.),
question? Journal of the Learning Sciences, 13(1), 105–114. Handbook of educational psychology (2nd ed., pp. 511–544).
Design-Based Research Collective. (2003). Design-based research: An Mahwah, NJ: Lawrence Erlbaum Associates.
emerging paradigm for educational inquiry. Educational Researcher, McKenney, S. E., & Reeves, T. C. (2013). Systematic review of design-
32(1), 5–8. based research progress: Is a little knowledge a dangerous thing?
Dewey, J. (1900). Psychology and social practice. Psychological Educational Researcher, 42(2), 97–100.
Review, 7, 105–124. *McKenney, S. E., & Reeves, T. C. (2012). Conducting educational
Drexler, W. (2010). The networked student: A design-based research design research. London: Routledge.
case study of student constructed personal learning environments in McKenney, S., van den Akker, J., & Nieveen, N. (2006). Design
a middle school science course. Unpublished doctoral dissertation, research from the curriculum perspective. In J. Van den Akker, K.
The University of Florida. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational
Edelson, D. (2002). Design research: What we learn when we engage in design research (pp. 67–90). London: Routledge.
design. The Journal of the Learning Sciences, 11(1), 105–121. Mills, G. E. (2002). Action research: A guide for the teacher researcher
Ejersbo, L., Engelhardt, R., Frølunde, L., Hanghøj, T., Magnussen, R., (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall.
& Misfeldt, M. (2008). Balancing product design and theoretical Mishra, P., & Koehler, M. (2006). Technological pedagogical content
insight. In A. Kelly, R. Lesh, & J. Baek (Eds.), The handbook of knowledge: A framework for teacher knowledge. Teachers College
design research methods in education (pp. 149–163). Mahwah, NJ: Record, 108(6), 1017–1054.
Lawrence Erlbaum Associates. Münsterberg, H. (1899). Psychology and life. Boston, MA: Houghton
*Firestone, W. A. (1993). Alternative arguments for generalizing from Mifflin.
data as applied to qualitative research. Educational Researcher, Newman, D. (1990). Opportunities for research on the organizational
22(4), 16–23. impact of school computers. Educational Researcher, 19(3), 8–13.
Fullan, M., & Pomfret, A. (1977). Research on curriculum and instruc- Oh, E. (2011). Collaborative group work in an online learning environ-
tion implementation. Review of Educational Research, 47(2), ment: A design research study. Unpublished doctoral dissertation,
335–397. The University of Georgia.
Glaser, R. (1976). Components of a psychology of instruction: Toward Oh, E., & Reeves, T. (2010). The implications of the differences
a science of design. Review of Educational Research, 46(1), 29–39. between design research and instructional systems design for educa-
Gravemeijer, K., & Cobb, P. (2006). Outline of a method for design tional technology researchers and practitioners. Educational Media
research in mathematics education. In J. V. D. Akker, K. Gravemeijer, International, 47(4), 263–275.
S. McKenney, & N. Nieveen (Eds.), Educational design research. *Penuel, W., Fishman, B., Cheng, B., & Sabelli, N. (2011). Organizing
London: Routledge. research and development at the intersection of learning, implemen-
Hall, G. E., Wallace, R. C., & Dossett, W. A. (1973). A developmental tation and design. Educational Researcher, 40(7), 331–337.
conceptualization of the adoption process within educational insti- Plomp, T., & Nieveen, N. (Eds.). (2009). An introduction to educational
tutions. Austin, TX: The Research and Development Center for design research: Proceedings of the seminar conducted at the East
Teacher Education. China Normal University, Shanghai. Enschede, The Netherlands:
Havelock, R. (1971). Planning for innovation through dissemination SLO—Netherlands Institute for Curriculum Institute.
and utilization of knowledge. Ann Arbor, MI: Center for Research Quintana, C., Reiser, B., Davis, E., Krajcik, J., Fretz, E., Duncan, R. G.
on Utilization of Scientific Knowledge. et al. (2004). A scaffolding design framework for software to
140 S. McKenney and T.C. Reeves

support science inquiry. The Journal of the Learning Sciences, Shavelson, R., Phillips, D., Towne, L., & Feuer, M. (2003). On the science
13(3), 337–386. of education design studies. Educational Researcher, 32(1), 25–28.
Raval, H. (2010). Supporting para-teachers in an Indian NGO: The Stokes, D. (1997). Pasteurs quadrant: Basic science and technological
plan-enact-reflect cycle. Unpublished doctoral dissertation, Twente innovation. Washington, DC: Brookings Institution Press.
University. Thomas, M. K., Barab, S. A., & Tuzun, H. (2009). Developing critical
Reeves, T. C. (2011). Can educational research be both rigorous and implementations of technology-rich innovations: A cross-case study
relevant? Educational Designer: Journal of the International of the implementation of Quest Atlantis. Journal of Educational
Society for Design and Development in Education, 1(4). Available Computing Research, 41(2), 125–153.
online at: http://www.educationaldesigner.org/ed/volume1/issue4/ van Aken, X. (2004). Management research based on the paradigm of
Reeves, T. C. (2006). Design research from the technology perspec- the design sciences: The quest for field-tested and grounded tech-
tive. In J. V. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen nological rules. Journal of Management Studies, 41(2),
(Eds.), Educational design research (pp. 86–109). London: 219–246.
Routledge. *van den Akker, J. (1999). Principles and methods of development
*Reinking, D., & Bradley, B. (2008). Formative and design experi- research. In J. van den Akker, R. Branch, K. Gustafson, N. Nieveen,
ments: Approaches to language and literacy research. New York, & T. Plomp (Eds.), Design approaches and tools in education and
NY: Teachers College Press. training (pp. 1–14). Dordrecht: Kluwer Academic Publishers.
Reynolds, R., & Caperton, I. (2011). Contrasts in student engagement, van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N.
meaning-making, dislikes, and challenges in a discovery-based pro- (Eds.). (2006a). Educational design research. London: Routledge.
gram of game design learning. Educational Technology Research van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N.
and Development, 59(2), 267–289. (2006b). Introducing educational design research. In J. Van den
Richey, R., & Klein, J. (2007). Design and development research. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.),
Mahwah, NJ: Lawrence Erlbaum Associates. Educational design research (pp. 3–7). London: Routledge.
Schoenfeld, A. H. (2009). Bridging the cultures of educational research Wang, F., & Hannafin, M. (2005). Design-based research and technol-
and design. Educational Designer: Journal of the International ogy-enhanced learning environments. Educational Technology
Society for Design and Development in Education, 1(2). Available Research and Development, 53(4), 5–23.
online at: http://www.educationaldesigner.org/ed/volume1/issue2/ Xie, Y., & Sharma, P. (2011). Exploring evidence of reflective thinking
article5/index.htm in student artifacts of blogging-mapping tool: A design-based
Schwarz, B. B., & Asterhan, C. S. (2011). E-moderation of synchro- research approach. Instructional Science, 39(5), 695–719.
nous discussions in educational settings: A nascent practice. The Yin, R. (1989). Case study research: Design and methods. Newbury
Journal of the Learning Sciences, 20(3), 395–442. Park, CA: Sage Publishing.