Received: 29 January 2018

| Accepted: 20 March 2018

DOI: 10.1111/eci.12931

PERSPECTIVE

Time to challenge the spurious hierarchy of systematic over

narrative reviews? https://www.ncbi.nlm.nih.gov/pmc/articles/
PMC6001568/pdf/ECI-48-na.pdf

Trisha Greenhalgh1 | Sally Thorne2 | Kirsti Malterud3

1
Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
2
School of Nursing, University of British Colombia, Vancouver, Canada
3
Research Unit for General Practice, Uni Research Health, Bergen, Norway

Correspondence
Trisha Greenhalgh, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK.
Email: [email protected]

Funding information
TG’s contribution was part-funded from the National Institute for
Health Research Biomedical Research Centre, Oxford, Grant BRC-
1215-20008.

1 | BACKGROUND methods section or statement on how studies were selected

and analysed (see for example7-9). These narrative reviews
Cynthia Mulrow’s important paper calling for literature typically draw on expert opinion by deliberately recruiting
reviews to be undertaken more systematically (and hence be leading names in the field (eg “The aim of this Commission
more informative and reliable) is now 30 years old.1 A recent is to provide the strongest evidence base through involve-
paper in BMC Medical Research Methodology compared the ment of experts from a wide cross-section of disci-
proportion of reviews that were systematic (as opposed to nar- plines. . .”—page 1953, emphasis added8). Reviews crafted
rative) in five leading biomedical journals.2 The authors found through the experience and judgement of experts are often
significant diversity: from New England Journal of Medicine viewed as untrustworthy (“eminence-based” is a pejorative
(0%) and Lancet (11%) to Annals of Internal Medicine (72%). term). Yet the classical definition of the EBM as “the con-
Systematic reviews were assumed by the authors to be supe- scientious, explicit, and judicious use of current best evi-
rior because they are (i) more likely to have a focused research dence . . .” (page 71, emphasis added)10 suggests a key role
question, (ii) more methodologically explicit and (iii) less for judgement in the selection and interpretation of evidence.
likely to be biased than narrative reviews. In short, there appears to be a growing divergence
This stance reflects the raison d’^etre of the Cochrane between the assumed “hierarchy” of evidence in secondary
Collaboration, whose use of explicit and auditable quality research, which defines systematic reviews as superior,11
criteria for undertaking systematic reviews has inspired a and what some leading academic journals view as a state-
weighty methodological handbook,3 numerous tools and of-the-art (that is, expert-led narrative) review. We believe
checklists4,5 and structured reporting criteria.6 There is this is partly because the systematic review format has been
strong emphasis on methodological reproducibility, with erroneously defined as a universal gold standard and partly
the implication that a different review team, using the same because the term “narrative review” is frequently misunder-
search criteria, quality checklists and synthesis tools, stood, misapplied and unfairly dismissed.
should obtain the same result.3 Systematic reviews in the Cochrane sense use a highly
Yet leading medical journals regularly publish clinical technical approach to identification, appraisal and synthesis
topic reviews that may lack a focused research question, of evidence and typically (although not invariably)

----------------------------------------------------------------------------------------------------------------------------------------------------------------------
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium,
provided the original work is properly cited and is not used for commercial purposes.
© 2018 The Authors. European Journal of Clinical Investigation published by John Wiley & Sons Ltd on behalf of Stichting European Society for Clinical Investigation
Journal Foundation.

Eur J Clin Invest. 2018;48:e12931. wileyonlinelibrary.com/journal/eci | 1 of 6

https://doi.org/10.1111/eci.12931
2 of 6
| GREENHALGH ET AL.

privilege randomised controlled trials or previous system-

atic reviews over other forms of evidence.11 This may be
Key point
entirely appropriate—especially when the primary purpose
is to answer a very specific question about how to treat a • Systematic reviews are generally placed above
particular disease in a particular target group. narrative reviews in an assumed hierarchy of sec-
But the doctor in the clinic, the nurse on the ward or the ondary research evidence
social worker in the community will encounter patients with a • We argue that systematic reviews and narrative
wide diversity of health states, cultural backgrounds, illnesses, reviews serve different purposes and should be
sufferings and resources.12 And those who gather around the viewed as complementary
policymaking table will find multiple calls on their attention— • Conventional systematic reviews address nar-
including burden of need, local availability of different treat- rowly focused questions; their key contribution is
ments, personal testimony, strength of public opinion and bud- summarising data
getary realities. To produce a meaningful synthesis of research • Narrative reviews provide interpretation and cri-
evidence relevant to such complex situations, the reviewer tique; their key contribution is deepening under-
must (i) incorporate a broad range of knowledge sources and standing
strategies for knowing and (ii) undertake multi-level interpre-
tation using creativity and judgement.12,13
We align with previous authors, who, drawing on exclusion criteria, creating tables of extracted data and
Wittgenstein, distinguish between puzzles or problems that mathematically summing effect sizes (rather than, for
require data (for which a conventional systematic review, example, with the level of critical analysis of the papers’
with meta-analysis where appropriate, may be the preferred unstated assumptions and discussion sections) has, we
methodology) and those that require clarification and believe, led to a proliferation of systematic reviews that
insight (for which a more interpretive and discursive syn- represent aggregations of findings within the narrow body
thesis of existing literature is needed).14,15 of work that has met the authors’ eligibility criteria.17-19
Below, we explore both strengths, limitations and con- Such studies may sometimes add value, especially when
ceptual confusions of systematic and narrative reviews. We additional meta-analysis confirms whether a clinically sig-
consider three questions: what makes a review systematic; nificant effect is or is not also statistically significant.20 But
what is a narrative review and whether these different kinds sometimes, the term “systematic review” allows a data
of review should be viewed as competing or complementary. aggregation to claim a more privileged position within the
knowledge hierarchy than it actually deserves.11
We acknowledge that the science of systematic review
2 | WHAT MAKES A REVIEW within the Cochrane and Campbell Collaborations is evolving
SYSTEMATIC? to embrace a wider range of primary studies and methodolo-
gies, with recommended procedures for sampling, assessment
The defining characteristic of a systematic review in the and synthesis of evidence compliant with the question asked
Cochrane sense is the use of a predetermined structured and the context explored. The adjective “systematic” is thus
method to search, screen, select, appraise and summarise coming to acquire a broader meaning in terms of the trans-
study findings to answer a narrowly focused research ques- parency and appropriateness of methods, rather than signify-
tion.3,16 Using an exhaustive search methodology, the ing strict adherence to a particular pre-defined tool or
reviewer extracts all possibly relevant primary studies, and checklist or a privileging of randomised trials (see for example
then limits the dataset using explicit inclusion and exclusion methodological work by Lewin et al,21 Petticrew et al22 and
criteria. The review focus is highly circumscribed and quality Pluye et al23-25). All these approaches, however, remain
criteria are tightly enforced. Typically, a body of hundreds or focused on answering a relatively narrow question that is pre-
thousands of potential studies identified in the initial search is defined at the outset and with a primary focus on the extrac-
whittled down to a mere handful before the reviewer even tion, tabulation and summation of empirical data.
begins to consider what they collectively mean.
The term “systematic” is thus by no means synonymous
with “high-quality”. Rather, it can be viewed as a set of 3 | WHAT IS A NARRATIVE
methodologies characterised by tight focus, exhaustive REVIEW?
search, high rejection-to-inclusion ratio and an emphasis on
technical rather than interpretive synthesis methods. A narrative review is a scholarly summary along with
The conflation of the quality of a review with the assid- interpretation and critique.26 It can be conducted using a
uousness of such tasks as searching, applying inclusion and number of distinctive methodologies. While principles and
GREENHALGH ET AL. | 3 of 6

procedures may diverge from the classic methodology of understanding not only of the topic in question but also of
systematic review, they are not unsystematic (in the sense the reasons why it has been studied in a particular way, the
of being ad hoc or careless), and may certainly be con- interpretations that have been variously made with respect
ducted and presented in a systematic way, depending on to what we know about it, and the nature of the knowledge
purpose, method and context. base that informs or might inform clinical practice.
Different kinds of reviews offer different kinds of truth: Because hermeneutic, realist and meta-narrative reviews
the conventional systematic review with meta-analysis have explicit methodologies and accepted standards and
deals in probabilistic (typically, Bayesian) truth; it is con- criteria for judging their quality,14,27,28 a minority of schol-
cerned mainly with producing generalisable “facts” to aid ars include such approaches within the (broadly defined)
prediction. The narrative review, in contrast, deals in plau- category of systematic reviews. However, we have had
sible truth. Its goal is an authoritative argument, based on experience of journal editors rejecting reviews based on
informed wisdom that is convincing to an audience of fel- these techniques on the grounds that they were “not sys-
low experts. To that end, the author of a narrative review tematic”. Also of note is the emergence of “how-to” guides
must authentically represent in the written product both the for narrative reviews, which (misleadingly in our view)
underpinning evidence (including but not limited to pri- exhort the reviewer to focus carefully on such tasks as
mary research) and how this evidence has been drawn upon starting with an explicit search strategy and defining strict
and drawn together to inform the review’s conclusions. inclusion and exclusion criteria for primary studies.35,36 In
A hermeneutic review takes as its reference point the other words, the boundaries between systematic and narra-
notion of verstehen, or the process of creating an interpre- tive reviews are both fuzzy and contested.
tive understanding.14 It capitalises on the continual deepen-
baixei
ing of insight that can be obtained by critical reflection on
particular elements of a dataset—in this case, individual 4 | SYSTEMATIC OR NARRATIVE
primary studies—in the context of a wider body of work. It OR SYSTEMATIC AND NARRATIVE?
may or may not define its reference body of studies using
systematic search methods and inclusion/exclusion criteria, The conflation of “systematic” with superior quality (and
but its primary focus is on the essential tasks of induction “narrative” with inferior quality) has played a major role
and interpretation in relation to the defined sample for the in the muddying of methodological waters in secondary
purpose of advancing theoretical understanding.17 A realist research. This implicit evidence hierarchy (or pyramid)
review considers the “generative causality,” in which par- elevates the mechanistic processes of exhaustive search,
ticular mechanisms (for example, peer influence) produce wide exclusion and mathematical averaging over the
particular outcomes (for example, smoking cessation) in thoughtful, in-depth, critically reflective processes of
some circumstances (for example, when societal disap- engagement with ideas. The emphasis on thinking and in-
proval of smoking is high) but not others (for example, in terpretation in narrative review has prompted some
cultures where smoking is still widely viewed as a mark of authors to use the term “evidence-informed” rather than
sophistication).27 A meta-narrative review maps the story- “evidence-based”15,37: the narrative review is both less
line of a research tradition over time.28 Shifting the focus and more than a methods-driven exercise in extracting
away from comparing findings of studies published at dif- and summating data.
ferent times, it orients critical reflection to discern how Training in systematic reviews has produced a genera-
ideas have waxed and waned within different scholarly tion of scholars who are skilled in the technical tasks of
communities at different points in the development of searching, sorting, checking against inclusion criteria, tabu-
thinking (see an early example of how the term “diffusion lating extracted data and generating “grand means” and
of innovations” was differently defined and explored in dif- confidence intervals.3 These skills are important, but as the
ferent academic disciplines29). recent article by Faggion et al illustrates, critics may incor-
Each of these forms of narrative review (along with rectly assume that they override and make redundant the
other specialist approaches to combining primary studies in generation of understanding. To the extent that the term
qualitative research30,31) reflects an explicit lens that is “systematic review” privileges only that which is common
expected to shape the understandings that will arise from in the findings amongst a rigidly defined subset of the
the review process, through analysis and synthesis processes available body of work, we risk losing sight of the marvel-
that may be highly systematic. Narrative reviews also lous diversities and variations that ought to intrigue us. In
include a number of more generic styles such as integra- excluding those aspects of scholarship, systematic reviews
tive32,33 and critical,34 the former being the approach gener- hold the potential to significantly skew our knowledge
ally taken by narrative reviews in clinical journals. All these landscape. While there are occasions when systematic
approaches play an important role in expanding our review is the ideal approach to answering specific types of
4 of 6
| GREENHALGH ET AL.

question, the absence of thoughtful, interpretive critical This principle was illustrated in a recent hermeneutic
reflection can render such products hollow, misleading and review of telehealth in heart failure by one of us.43 It iden-
potentially harmful. tified 7 systematic reviews of systematic reviews, 32 sys-
The argument that systematic reviews are less biased tematic reviews (including 17 meta-analyses) covering
than narrative reviews begs the question of what we mean hundreds of primary studies, as well as six mega-trials—
by bias. Bias is an epidemiological construct, which refers almost all of which had concluded that more research (ad-
to something that distorts the objective comparisons dressing the same narrow question with yet more ran-
between groups.20 It presupposes the dispassionate, instru- domised trials intended to establish an effect size for
mental and universal “view from nowhere” that has long telehealth) was needed. The hermeneutic approach revealed
defined the scientific method.38 When we speak of inter- numerous questions that had remained under-explored as
pretation, we refer to an analysis that is necessarily per- researchers had pursued this narrow question—including
spectival, with the interpreter transparently positioned in the complex and changing nature of the co-morbidities and
order that the reader can understand why this particular per- social determinants associated with heart failure, the varied
spective, selection process and interpretive methodology experiences and priorities of patients with heart failure, the
was selected in relation to the question at hand.14,17,29,37,39 questionable nature of up-titration as a guiding principle in
Systematic and transparent reflection upon and sharing of heart failure management, and the numerous organisational,
such aspects of the research process adds to the scientific regulatory and policy-level complexities associated with
quality of interpretive research. introducing telehealth programmes. The review concluded
The question of whether “systematic” review techniques that: “The limited adoption of telehealth for heart failure
can eliminate bias in secondary research is in any case has complex clinical, professional and institutional causes,
questionable. The privileging of freedom from bias over which are unlikely to be elucidated by adding more ran-
relevance of question and findings wrongly assumes that domised trials of technology-on versus technology-off to
how the topic is framed, and which questions should be an already-crowded literature. An alternative approach is
explored is somehow self-evident. A recent review of sys- proposed, based on naturalistic study designs, application
tematic reviews generated by a national knowledge centre of social and organisational theory, and co-design of new
to inform policymaking in Norway showed that in most service models based on socio-technical principles” (page
cases, the evidence base addressed only a fraction of rele- 156).
vant policy questions.40 More generally, there is growing
evidence that the science of systematic reviews is becom-
ing increasingly distorted by commercial and other con- 5 | CONCLUSION
flicts of interest, leading to reviews, which—often despite
ticking the boxes on various quality checklists—are unnec- As many authors and journal editors are well aware, the
essary, misleading or partisan.19,41 The holy grail of a narrative review is not a poor cousin of the systematic
comprehensive database of unambiguous and unbiased evi- review but a different and potentially complementary form
dence summaries (in pursuit of which the Cochrane Col- of scholarship.22,44 Nevertheless, the simplistic hierarchy
laboration was founded42) continues to recede into the “systematic review good; narrative review less good” per-
future. sists in some circles. The under-acknowledged limitations
A legitimate criticism of narrative reviews is that they of systematic reviews, along with missed opportunities for
may “cherry pick” evidence to bolster a particular perspec- undertaking and using narrative reviews to extend under-
tive. But this must be weighed against the counter-argu- standing within a field, risks legitimising and perpetuating
ment that the narrative reviewer selects evidence a narrow and unexciting research agenda and contributing
judiciously and purposively with an eye to what is relevant to research waste. We call upon policymakers and clini-
for key policy questions—including the question of which cians (who seek to ensure that their decisions are evidence-
future research programmes should be funded. Whilst we based, but who may have been seduced by a spurious hier-
accept that narrative reviews can be performed well or archy of secondary evidence) and on research commission-
badly, we believe the undervaluing of such reviews is a ers (whose decisions will shape the generation of the future
major contributor to research waste. In the absence of an evidence base) to re-evaluate the low status currently
interpretive overview of a topic that clearly highlights the afforded to narrative reviews.
state of knowledge, ignorance and uncertainty (explaining
how we know what we know, and where the intriguing
AUTHORS’ CONTRIBUTIONS
unanswered questions lie), research funding will continue
to be ploughed into questions that are of limited impor- TG was invited to submit a paper on a topic of her choice
tance, and which have often already been answered.40 to EJCI by the editor. She suggested this topic to ST and
GREENHALGH ET AL. | 5 of 6

KM and wrote an initial outline for the paper. All authors 20. Egger M, Smith GD, Altman D. Systematic Reviews in Health
then contributed iteratively and equally to the development Care: Meta-Analysis in Context. Oxford: Wiley; 2008.
of ideas and refinement of the paper. 21. Lewin S, Glenton C, Oxman AD. Use of qualitative methods
alongside randomised controlled trials of complex healthcare
interventions: methodological study. BMJ. 2009;339:b3496.
REFERENCES 22. Petticrew M, Rehfuess E, Noyes J, et al. Synthesizing evidence
on complex interventions: how meta-analytical, qualitative, and
1. Mulrow CD, Thacker SB, Pugh JA. A proposal for more informative
mixed-method approaches can contribute. J Clin Epidemiol.
abstracts of review articles. Ann Intern Med. 1988;108:613-615.
2013;66:1230-1243.
2. Faggion CM, Bakas NP, Wasiak J. A survey of prevalence of
23. Hong QN, Pluye P, Bujold M, Wassef M. Convergent and
narrative and systematic reviews in five major medical journals.
sequential synthesis designs: implications for conducting and
BMC Med Res Methodol. 2017;17:176.
reporting systematic reviews of qualitative and quantitative evi-
3. Higgins JP, Green S. Cochrane Handbook for Systematic
dence. Syst Rev. 2017;6:61.
Reviews of Interventions, vol. 4. Chichester, UK: John Wiley &
24. Pluye P. Mixed kinds of evidence: synthesis designs and critical
Sons; 2011.
appraisal for systematic mixed studies reviews including qualita-
4. Shea BJ, Grimshaw JM, Wells GA, et al. Development of
tive, quantitative and mixed methods studies. Evid-Based Med.
AMSTAR: a measurement tool to assess the methodological qual-
2015;20:79.
ity of systematic reviews. BMC Med Res Methodol. 2007;7:10.
25. Pluye P, Hong QN, Bush PL, Vedel I. Opening-up the definition
5. Whiting P, Savovic J, Higgins JP, et al. ROBIS: a new tool to
of systematic literature review: the plurality of worldviews,
assess risk of bias in systematic reviews was developed. J Clin
methodologies and methods for reviews and syntheses. J Clin
Epidemiol. 2016;69:225-234.
Epidemiol. 2016;73:2-5.
6. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement
26. MacLure M. ‘Clarity bordering on stupidity’: where’s the quality
for reporting systematic reviews and meta-analyses of studies that
in systematic review? J Edu Policy. 2005;20:393-416.
evaluate health care interventions: explanation and elaboration.
27. Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R.
PLoS Med. 2009;6:e1000100.
RAMESES publication standards: realist syntheses. BMC Med.
7. Pedersen BK. Anti-inflammatory effects of exercise: role in dia-
2013;11:21.
betes and cardiovascular disease. Eur J Clin Invest. 2017;47:
28. Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R.
600-611.
RAMESES publication standards: meta-narrative reviews. BMC
8. Williams R, Aspinall R, Bellis M, et al. Addressing liver disease
Med. 2013;11:20.
in the UK: a blueprint for attaining excellence in health care and
29. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O,
reducing premature mortality from lifestyle issues of excess con-
Peacock R. Storylines of research in diffusion of innovation: a
sumption of alcohol, obesity, and viral hepatitis. Lancet.
meta-narrative approach to systematic review. Soc Sci Med
2014;384:1953-1997.
2005;61:417-430.
9. Cannon JW. Hemorrhagic Shock. N Eng J Med. 2018;378:370-379.
30. Paterson BL, Canam C. Meta-study of Qualitative Health
10. Sackett DL, Rosenberg WM, Gray JM, Haynes RB, Richardson
Research: A Practical Guide to Meta-analysis and Meta-synthesis,
WS. Evidence based medicine: what it is and what it isn’t. BMJ.
vol. 3. London, UK: Sage; 2001.
1996;312:71-72.
31. Thorne S. Metasynthetic madness: what kind of monster have we
11. Alper BS, Haynes RB. EBHC pyramid 5.0 for accessing preap-
created? Qual Health Res. 2017;27:3-12.
praised evidence and guidance. BMJ. 2016;21:123-125.
32. Whittemore R, Knafl K. The integrative review: updated method-
12. Whitty CJ. What makes an academic paper useful for health pol-
ology. J Adv Nurs. 2005;52:546-553.
icy? BMC Med. 2015;13:301.
33. Torraco RJ. Writing integrative literature reviews: guidelines and
13. Malterud K. The art and science of clinical knowledge: evidence
examples. Hum Resour Dev Rev. 2005;4:236-367. INSERT
beyond measures and numbers. The Lancet. 2001;358:397-400.
34. Saunders MN, TRojon C. On the attributes of a critical literature
14. Boell SK, Cecez-Kecmanovic D. A hermeneutic approach for
review. Coaching. 2011;4:158-162.
conducting literature reviews and literature searches. CAIS.
35. Randolph JJ. A guide to writing the dissertation literature review.
2014;34:12.
PARE. 2009;14:1-13.
15. Nevo I, Slonim-Nevo V. The myth of evidence-based practice:
36. Byrne JA. Improving the peer review of narrative literature
towards evidence-informed practice. Br J Soc Work.
reviews. Res Integr Peer Rev. 2016;1:12.
2011;41:1176-1197.
37. Thorne S, Sawatzky R. Particularizing the general: Sustaining
16. Morton S, Berg A, Levit L, Eden J. Finding What Works in
theoretical integrity in the context of an evidence-based practice
Health Care: Standards for Systematic Reviews. Washington,
agenda. Adv Nurs Sci. 2014;37:5-18.
DC: National Academies Press; 2011.
38. Merton RK. Science and the social order. Philos Sci. 1938;5:321-337.
17. Dixon-Woods M, Cavers D, Agarwal S, et al. Conducting a criti-
39. Webster J, Watson RT. Analyzing the past to prepare for the
cal interpretive synthesis of the literature on access to healthcare
future: writing a literature review. MIS Q 2002;26:xiii-xxiii.
by vulnerable groups. BMC Med Res Methodol. 2006;6:35.
40. Malterud K, Bjelland AK, Elvbakken KT. Evidence-based
18. Greenhalgh T. Outside the box: why are cochrane reviews so
medicine–an appropriate tool for evidence-based health policy? A
boring? Br J Gen Pract. 2012;62:371-371.
case study from Norway. Health Res Policy Syst. 2016;14:15.
19. Ioannidis J. The mass production of redundant, misleading, and
41. Greenhalgh T, Howick J, Maskrey N. Evidence based medicine:
conflicted systematic reviews and meta-analyses. Milbank Q.
a movement in crisis? BMJ. 2014;348:g3725.
2016;94:485-514.
6 of 6
| GREENHALGH ET AL.

42. Levin A. The cochrane collaboration. Ann Intern Med.

2001;135:309-312.
How to cite this article: Greenhalgh T, Thorne S,
43. Greenhalgh T, Shaw S, A’Court C. Understanding heart failure;
explaining telehealth–a hermeneutic systematic review. BMC Car-
Malterud K. Time to challenge the spurious
diovasc Disord. 2017;17:156. hierarchy of systematic over narrative reviews?. Eur
44. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Syn- J Clin Invest. 2018;48:e12931. https://doi.org/
thesising qualitative and quantitative evidence: a review of possi- 10.1111/eci.12931
ble methods. J Health Serv Res Policy. 2005;10:45-53.