Journal of Clinical Epidemiology 152 (2022) 288e294

COMMENTARY

Commentary: collaborative systematic review may produce and share

high-quality, comparative evidence more efficiently
Jill A Hayden*, for the BACK Evidence Collaboration - Collaborative Review Working Group
Department of Community Health & Epidemiology, Dalhousie University, Halifax, Canada
Accepted 22 September 2022; Published online 28 September 2022

Abstract
Systematic reviews are necessary to synthesize available evidence and inform clinical practice and health policy decisions. There has
been an explosion of evidence available in many fields; this makes it challenging to keep evidence syntheses up to date and useful. Compar-
ative effectiveness systematic reviews are informative; however, producing these often-large reviews bring intense time and resource de-
mands. This commentary describes the implementation of a systematic review using a collaborative model of evidence synthesis. We are
implementing the collaborative review model to update a large Cochrane review investigating the efficacy and comparative effectiveness of
the design, delivery, and type of exercise treatment for people with chronic low-back pain. Three key benefits of the collaborative review
model for evidence synthesis are (1) team coordination and collaboration, (2) quality control measures, and (3) advanced comparative and
other analyses. This new collaborative review model is developed and implemented to produce and share high-quality, comparative evi-
dence more efficiently while building capacity and community within a research field. Ó 2022 Elsevier Inc. All rights reserved.

Keywords: Collaborative review model; Network meta-analysis; Systematic review; Evidence synthesis; Cochrane review; Clinical trials

1. Collaborative review model commentary and clinicians’ decisions about patient care as well as health
policies. However, current approaches to systematic review
Well-conducted evidence syntheses are essential to
production result in a fragmented evidence base. While ev-
inform decision-making. The body of scientific literature idence synthesis infrastructure exists within organizations
in the health sciences is growing exponentially [1].
such as Cochrane, there are acknowledged limitations with
Health-care and policy decision-makers need the best qual-
the current approach to systematic review.
ity evidence to provide the best care. However, it is chal-
Systematic reviews are most commonly produced on
lenging to stay informed and up to date. Systematic
highly-focused topics (i.e., 1 treatment vs. control for 1
reviews use methods to transparently synthesize and
condition) and, as a result, provide fragmented evidence
interpret all evidence relevant to a specific research ques-
with gaps for decision-making, especially in topic areas
tion [2]. Described as the ‘‘cornerstone of evidence-based
where a breadth of information exists [4,5]. For example,
medicine’’ [3], systematic reviews are at the top of the ev- a systematic review might examine the efficacy of core
idence hierarchy; they inform clinical practice guidelines
strengthening exercises for the treatment of chronic low-
back pain but not consider how effective core strengthening
is compared to other exercise types. The need for system-
Funding: The Canadian Institutes of Health Research provided funding atic reviews that compare multiple competing treatments,
for this project (Project Grant Competition, PJT-173478). MLF holds a Na- for example, with large network meta-analyses [6], directly
tional Health and Medical Research Council of Australia Investigator conflicts with the intense time and resource demands of
Fellowship.
Conflict of interest: Jill A Hayden and Collaborative Review Working
current production models [7e9].
Group members report no other known competing interests. Most systematic reviews are conducted by small teams
Data sharing: Not relevant. with tools and processes developed for the specific systematic
* Corresponding author. Department of Community Health & Epide- review. There is considerable duplication of effort with over-
miology, Dalhousie University, 5790 University Avenue, Room 404, Hal- lapping systematic reviews and inclusion of the same studies
ifax, Nova Scotia, Canada B3H 1V7. Tel.: þ1 902 494 4046; fax: þ1 902
494 1597.
in multiple systematic reviews on similar topics. Siontis et al.
E-mail address: [email protected]. found that, on average, 67% of meta-analyses published in
https://doi.org/10.1016/j.jclinepi.2022.09.013
0895-4356/Ó 2022 Elsevier Inc. All rights reserved.
 J.A. Hayden / Journal of Clinical Epidemiology 152 (2022) 288e294 289

developed this collaborative review approach alongside dis-

What is new? cussions and early implementation of living systematic re-
views. While living systematic reviews focus on frequent
 The collaborative review model has 3 key features
updates of evidence (e.g., developed for fields where rec-
and associated benefits: (1) team coordination and
ommendations for best treatments change quickly), the
collaboration, (2) quality control measures, and (3)
collaborative review model focuses on coordination, collab-
advanced comparative and other analyses.
oration, and data and resource sharing across related re-
 Further benefits of the collaborative review model views (e.g., developed for fields where there is a large
include reducing research waste, sharing resources, volume of research produced on a specific health condi-
training the next generation of reviewers, and tion). However, because large teams are required with both
building research capacity and community in the models, similar structures and efficiency tools may be used
research field. with both the collaborative review model and living sys-
tematic reviews.
 By collecting a large, robust data set and providing
Three key features and benefits of the collaborative re-
evidence across interventions, this collaborative re-
view model are (1) team coordination and collaboration,
view model allows preplanning and leveraging of
(2) quality control measures, and (3) advanced comparative
data and resources for additional topic area and
and other analyses (Table 1).
methodological investigations.
1.1.1. Team coordination and collaboration
In the collaborative review model, a large coordinated
team (the ‘‘network’’) works collaboratively and in smaller
2010 had at least one other overlapping meta-analysis
teams to complete subreviews within an overarching topic
covering the same comparisons, indications, and outcomes
area, using homogeneous methods and tools and by sharing
[10]. While some replication is useful as a validation for the
skills and resources [11]. A clear governance structure pro-
scientific community, efforts of searching, selecting, assess-
vides a fair process, oversight, and prioritization of research
ing, and extracting data from the same studies represent
efforts. Collaborative input on method decisions is sup-
countless hours of research waste if these data are not reus-
ported by extensive guidance documents that are shared
able or even publicly accessible. Furthermore, such a dupli-
across the network and multifaceted strategies for effective
cated effort creates uncertainty for clinical decision-making
communication. The opportunity to share concerns over
when systematic review teams in the same topic area use
study data and trustworthiness of included trials will also
different methods (with varying levels of transparency),
help to ensure that there is a common understanding of
leading to contradicting conclusions about the effectiveness
these issues and that a common approach to addressing
of specific treatments.
such concerns is adopted across the subreview teams. Coor-
Our solution to the challenge is a new coordinated model
dinated literature searching across a broader topic can be
of systematic review that more efficiently produces high-
more sensitive, with no duplication in search execution
quality, comparative evidence that we call a ‘‘collaborative
and study screening (i.e., all teams work together to screen
review model.’’ This collaborative review model enables
1 broad search return, instead of each team screening their
the division of a large systematic review topic into smaller,
own unique search return). Data-collection forms are devel-
focused ‘‘subreviews,’’ without the loss of the ability to
oped once and used by all teams, with a consistent collec-
conduct important overarching comparative and multivari-
tion of important variables across the overarching and all
able analyses. We are implementing the collaborative re-
subreviews, followed by consistent handling of data and an-
view model in the ongoing Cochrane review ‘‘Exercise
alytic methods (e.g., data cleaning, checking, and handling
therapy for chronic low back pain: a network meta-anal-
of missing data and analyses). The network, which is made
ysis’’, with 9 overlapping Cochrane reviews, alongside an
up of international senior and early career researchers, cre-
evidence update for a clinical practice guideline.
ates a ‘‘community of practice’’ environment with training,
Complete information about the methods used in the
mentorship, and open data access. The regular interaction
conduct of the systematic review using the collaborative re-
of network members builds capacity and facilitates coordi-
view model will be available in our Cochrane protocol
nated research planning in the field.
(Hayden 2022).
1.1.2. Quality control measures
1.1. What is the collaborative review model and what
The collaborative review model does not change the
are the potential benefits?
methods of systematic review conduct. The primary shift
The collaborative review model draws on strengths, con- is who conducts the review steps, the technology used to
cepts, and tools from other evidence synthesis approaches, facilitate the collaboration, and additional components
including comparative effectiveness reviews, Cochrane (Fig. 1). The overarching review and subreviews use robust
overviews, and living systematic reviews. We originally methods of evidence synthesis, applying standard
290 J.A. Hayden / Journal of Clinical Epidemiology 152 (2022) 288e294

Table 1. Summary of expected impacts of the collaborative review model for the Cochrane review, ‘‘Exercise therapy for chronic low back pain: a
network meta-analysis’’
User Systematic review challenges Expected impact of the collaborative review model
SR infrastructure Duplication and waste across reviews due to Without this model, the ‘‘exercise for chronic LBP’’ review
organizations small-scope topics would be divided to be feasible. The alternative, many
focused reviews with no overarching coordinated review,
would result in hours of duplication (search, selection, data
extraction, ROB); waste from unclassified trials
Duplication outside of Cochrane magnified with Ability to keep current should reduce the substantial number of
outdated Cochrane reviews overlapping, independently conducted reviews published
SRs and updates are time-intensive, particularly Coordination/collaboration shares the demands of evidence
for large reviews synthesis across larger teams for an efficient, quality
approach
Researchers SRs completed by small, siloed teams Field and review team benefit from expertise of large
international collaborators, capacity building with new
reviewers with training and mentorship
SRs require specialized training, expertise,
awareness of new methods
Single use of SR data is not efficient; Benefit the research field with data sharing and advancing
duplication; inconsistent methods priority research in the field
SR evidence users Inadequate reporting of RCTs exclude study Trialists benefit from clear messaging about what data points
findings in SRs are required to be included in reviews
Typical ‘‘pairwise’’ SR methods (e.g., A vs. B or Comparative effectiveness analyses: network meta-analysis,
A vs. nothing) do not recognize hierarchical meta-regression will allow assessment of relevant, complex
evidence treatment characteristics
Inconsistent methods (PICO, timing) impair Provide comparable, high-quality data for clinical
synthesis for post-hoc ‘‘overviews’’ of SRs management; eliminate the need for comparative
effectiveness reviews starting from scratch

Abbreviations: SR, systematic review; LBP, low-back pain; ROB, risk of bias; RCT, randomized controlled trial.

systematic review methods advocated by Cochrane [12], level of the subreviews and advanced analyses with com-
and reported following Preferred Reporting Items for Sys- parison of treatment effects for specific intervention char-
tematic Reviews and Meta-Analyses guidelines [13]. For acteristics and types at the level of the overarching review.
implementation of our collaborative review model, our When appropriate, network meta-analysis (statistical anal-
methods will be assessed and approved by the Cochrane ysis) will enable multiple treatments and design/delivery
Musculoskeletal group editorial board as well as our guide- characteristics to be compared and ranked using both
line stakeholders to ensure trust in the resulting data. direct and indirect evidence [14], making full use of avail-
Processes that we have incorporated into the collabora- able study data that would otherwise be excluded from
tive review model for quality control include (1) training traditional pairwise comparisons. The collaborative
and testing of new reviewers with detailed guidance docu- network and resulting output will also support supple-
ments for consistent decisions and data extraction, (2) a re- mental and methods projects. We aim to encourage
view management software program that includes data improved conduct and reporting in primary studies to
rules to limit data-extraction errors, (3) central record keep- further reduce research waste.
ing and reporting using best practices, (4) survey-based
trial author data-checking procedure, and (5) comprehen-
sive data-cleaning and validation procedures prior to 1.2. Setup and structure of the collaborative review
analyses. model in practice
Our collaborative review model, which we have success-
1.1.3. Advanced comparative and other analyses fully implemented to the step of data cleaning (Fig. 1), is
Comprehensive data collection, including all items rele- facilitated by a coordinating team (or ‘‘central team’’).
vant to the overview, subreviews, and decision-maker The central team organizes the overall conduct of the sys-
questions and collaborative collection using standardized tematic review, including training, planning, and facilita-
data forms and guidance, has resulted in a large and robust tion of meetings; oversight of the search process;
data set that provides evidence across interventions. Our development of screening forms, data-collection forms,
data collection was designed to allow for synthesis at the and companion guidance materials; and workflow. In our
 J.A. Hayden / Journal of Clinical Epidemiology 152 (2022) 288e294 291

Figure. Flow of a systematic review highlighting the steps and main features of traditional systematic reviews and additional components of our
community systematic review. The first 6 steps, indicated in green, have been completed; yellow indicates currently in progress; orange indicates
the 2 final steps remaining. CSR, collaborative systematic review; SR, systematic review; DE, data extraction; NMA, network meta-analysis. (For
interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

review implementation, this has required dedication of mul- In our systematic review implementation, we developed
tiple team members. All interested review teams within the and administered a mandatory training module to calibrate
broad scope of the topic area are identified and invited to all reviewers and ensure consistent decisions and extrac-
participate. For our review implementation, a team tasked tions. The training module included training videos and a
to synthesize evidence on the topic for a clinical practice ‘‘test project’’ including screening, data extraction, and
guideline also joined the collaboration to contribute to risk-of-bias assessment. Reviewers were provided feedback
and benefit from the output. and met together to discuss data extracted and reach
292 J.A. Hayden / Journal of Clinical Epidemiology 152 (2022) 288e294

consensus. Thirty-five team members, including 31 affili- 1.4. Contributions and recognition
ated with subreviews and the guideline team and 4 central
The collaborative review model includes a large interna-
team members, completed the training and contributed to
tional team of researchers and research personnel at various
review screening and data extraction for results from the
career stages. Some collaborative review team members are
first search between February and June 2022. We anticipate
systematic review leads, while others are research staff and
this team will continue to work together on subsequent
trainees. The model offers opportunities for early career re-
search updates.
searchers to receive mentorship in the systematic review
The collaborative review model is governed by a steer-
process by senior review leads and steering committee
ing committee of 13 international leaders in evidence syn-
thesis and knowledge translation, with systematic review members through the conduct of their review. All team
members have the opportunity to contribute to the over-
authors, content area and methods experts, health-care
arching review, their own subreview(s), and additional sup-
practitioners, patient advisors, and guideline representa-
plemental or methodological projects which take advantage
tives, who monitor and guide the initiative. The committee
of the expansive and rigorously developed data set created
has informed the development of processes for governance
by the collaborative review model.
(e.g., working groups) and the prioritization and definitions
of interventions and outcomes through regular online sur-
veys and meetings.
1.5. Potential challenges
We have identified 3 related primary challenges in the im-
plementation of the collaborative review model: (1) cost, (2)
team motivation, and (3) acknowledgment of contributions.
1.3. Organization and tools
Despite their clinical value, in our experience, systematic re-
Unique to the collaborative review model is the large views are often unfunded research work. The collaborative
number of distinct reviewers contributing to the over- review model requires funding for at least 1 dedicated full-
arching and subreview projects. Once the full team iden- time team member to administer the model, in addition to
tifies all trials relevant for inclusion in the overarching the support of review team members (usually in-kind from
review, trials are allocated to subreview teams for data their home institutions). Personnel funding for evidence syn-
extraction based on (1) included treatments and (2) equi- thesis is an important factor to support evidence-informed
table division of labor. By working together, each trial is decision-making. Administration and project management
extracted by only 1 team, instead of every team where a of the process represent key parts of traditional systematic
relevant treatment group is included. Once data extraction review production time and costs [16]. They are particularly
is complete, the data for all included trials are easily parti- instrumental in collaborative systematic review, where hav-
tioned off according to the systematic review eligibility ing a centralized, experienced project management is likely
criteria of each subreview. We developed extensive guid- to bring efficiencies and improve effectiveness.
ance documentation to support the review title and abstract Funding is also required to cover the cost of licensing if
screening, full-text screening, data extraction, and risk-of- the initiative uses a for-cost electronic systematic review
bias assessment to support a consistent approach. Guidance tool. We used DistillerSR and found that it met our needs,
documentation was linked directly to the electronic forms but cheaper or no-cost options are available (e.g., Covi-
used to conduct screening and data extraction and were dence, Veritas Health Innovation, Melbourne, Australia;
embedded in data-extraction form instructions and Microsoft Access, Microsoft Corporation, Redmond, WA)
prompts. if funds are limited. The lack of access to a software pro-
This collaborative review model has been supported by 2 gram tailored to collaborative and systematic reviews has
key electronic resources: an online meeting forum and a been identified as a barrier to participation in systematic re-
web-based systematic review software program. The cen- views [16]; however, shared use of expertise and tools with
tral team has employed Microsoft Teams (Microsoft Corpo- central funding may be more likely with a collaborative
ration, Redmond, WA) to facilitate regular video meetings model. We think that shared access to systematic review
and chat forums to address the questions of collaborative tools facilitated participation of team members, including
review team members in a fast and efficient way. We have team members from low- to middle-income countries.
used DistillerSR (DistillerSR Inc, Ottawa, Canada), a web- The large number of contributing team members substan-
based systematic review software program designed to tially reduced the project duration (although perhaps not
facilitate collaboration [15]. Within DistillerSR, all team total hours) from what was feasible with only our central
members work within 1 project to screen citations from team. We think that this makes the model particularly use-
the broad search, extract study data, and conduct risk-of- ful for topic areas where rapid evidence synthesis is
bias assessment using the pretested, validated forms. These important.
2 electronic resources, working together, allow reviewers in A second challenge that we anticipated was to keep the
different locations to work together seamlessly. individual network team members motivated and engaged
 J.A. Hayden / Journal of Clinical Epidemiology 152 (2022) 288e294 293

in the conduct of the individual reviews. With a large team, pain, on which Dr. Jill Hayden is also the lead author.
there are likely to be team members who lose ability, inter- Dr. Hayden (nominated guarantor) is a back pain and sys-
est, or motivation to engage. In our collaborative review tematic review researcher and lead researcher within the
model, we hold regular meetings and send email reminders BACK program. Dr. Hayden developed the concept and
and feedback. The central team monitors team member ac- wrote the draft manuscript. The Back Evidence Collabora-
tivity to offer support and encouragement and, when neces- tioneCollaborative Review Working Group includes cen-
sary, reassign the work to another team to support the tral team members (Rachel Ogilvie, Sareen Singh, Shazia
project timeline. All team members starting study screening Kashif), steering committee members (Jan Hartvigsen,
and data extraction were engaged throughout these steps. Chris Maher, Andrea Furlan, Toby Lasserson, Peter Tug-
Only 1 individual who initially volunteered to participate well, Maurits van Tulder, Amir Qaseem, Manuela Ferreira,
opted out due to the time required to complete the training Rachelle Buchbinder), Cochrane subreview leads (L. Susan
module. Wieland, Fabianna Moraleida, Bruno T Saragiotto, Tie
A third challenge is to ensure a transparent and fair pro- Parma Yamato, Annemarie de Zoete, Kasper B€ulow, Lisan-
cess for authorship and acknowledgment on the multiple dra Almeida de Oliveira, Geronimo Bejarano), and
outputs, given the large number of contributors [17]. Our clinical practice guideline evidence update lead (Carol
approach has been to reach consensus on a general process Cancelliere).
for determining authorship on collaborative outputs prior to
starting and to maintain open communication about this
with all team members as projects evolve. We have not
yet experienced any concerns with these practices. Acknowledgments
The authors appreciate the contributions of Leah Bou-
1.6. Opportunities for the collaborative review model los and Kristy Hancock, Evidence Synthesis Coordinators
at the Maritime SPOR Support Unit for literature search
This collaborative review model can have a potential
development and implementation, Samuel A Stewart for
impact beyond reviewing the topic area evidence. We have
statistical guidance, and additional central and subreview
already observed benefits of expert discussions to improve
team members contributing to study screening, data
research methods. Furthermore, our prioritized involvement
extraction, and critical appraisal: Nora Bakaa, Jennifer
of early career researchers supports education and succes-
Cartwright, Pedro Isaac Santos Chaves, Gaelan Connell,
sion. Our intention is to expand the collaborative review
Cristiano Costa, Ben Csiernik, Stephanie Di Pelino, Ju-
model to encompass the broader low-back pain field, to
nior Vitorino Fandim, Shireen Harbin, Dr. Wilhelmina IJ-
make resources and tools publicly available, to share data,
zelenberg, Carsten Bogh Juhl, Mariana Leite, Alanna
and to provide clear and consistent evidence for clinical
MacDonald, Devin Manning, Daniele Sirineu Pereira,
practice guidelines. We also hope to contribute to a sustain-
Dr. Diego Roger-Silva, Heather Shearer, Danielle South-
able increase in the forward momentum prioritizing useful
erst, Maria N Wilson, Jessica Wong, Leslie Verville,
clinical and methodologic research evidence by reducing
and Hainan Yu.
duplication of efforts currently seen in our field. Our steer-
ing committee has, so far, approved 2 supplemental projects
that will use, and supplement, the data from our collabora-
tive systematic review. We hope to encourage further References
collaboration, use, and growth of this data set with our plan [1] Bornmann L, Mutz R. Growth rates of modern science: a bibliometric
to make the meta-data from this initiative publicly analysis based on the number of publications and cited references.
available. JASIST 2015;66:2215e22.
We suggest that experts in other fields of study also [2] Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ,
et al. Cochrane Handbook for Systematic Reviews of Interventions.
develop initiatives to apply this collaborative review 2nd ed. Chichester, UK: John Wiley & Sons; 2019.
model. This will be particularly useful in fields where the [3] Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E.
volume of evidence is substantial, research questions can Systematic review automation technologies. Syst Rev 2014;3:74.
be broken down and addressed by related systematic re- [4] Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JP, Mavergames C,
views, where they draw on overlapping studies, and a more et al. Living systematic reviews: an emerging opportunity to narrow
the evidence-practice gap. PLoS Med 2014;11:e1001603.
comprehensive understanding would benefit clinical deci- [5] Crequit P, Trinquart L, Yavchitz A, Ravaud P. Wasted research when
sion-making. systematic reviews fail to provide a complete and up-to-date evidence
synthesis: the example of lung cancer. BMC Med 2016;14:8.
[6] Ades AE, Caldwell DM, Reken S, Welton NJ, Sutton AJ, Dias S. Ev-
idence synthesis for decision making 7: a reviewer’s checklist. Med
CRediT authorship contribution statement Decis Making 2013;33:679e91.
[7] Tricco AC, Brehaut J, Chen MH, Moher D. Following 411 Cochrane
This article describes the approach in conduct of the Co- protocols to completion: a retrospective cohort study. PLoS One
chrane review of exercise therapy for chronic low-back 2008;3:e3684.
294 J.A. Hayden / Journal of Clinical Epidemiology 152 (2022) 288e294

[8] Tricco AC, Moher D, Chen MH, Daniel R. Factors predicting [13] Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC,
completion and time to publication of Cochrane reviews. Open Ioannidis JPA, et al. The PRISMA statement for reporting systematic
Med 2009;3:e210e4. reviews and meta-analyses of studies that evaluate health care interven-
[9] Turner T, Green S, Tovey D, McDonald S, Soares-Weiser K, tions: explanation and elaboration. J Clin Epidemiol 2009;62:e1e34.
Pestridge C, et al. Producing Cochrane systematic reviewsda quali- [14] Riley RD, Jackson D, Salanti G, Burke DL, Price M, Kirkham J, et al.
tative study of current approaches and opportunities for innovation Multivariate and network meta-analysis of multiple outcomes and
and improvement. Syst Rev 2017;6:147. multiple treatments: rationale, concepts, and examples. BMJ 2017;
[10] Siontis KC, Hernandez-Boussard T, Ioannidis JP. Overlapping meta- 358:j3932.
analyses on the same topic: survey of published studies. BMJ 2013; [15] DistillerSR. Ottawa, ON: Evidence Partners; 2020.
347:f4501. [16] Nussbaumer-Streit B, Ellen M, Klerings I, Sfetcu R, Riva N,
[11] Charidimou A, Soo Y, Heo JH, Srikanth V, for the META- Mahmic-Kaknjo M, et al. Resource use during systematic review pro-
MICROBLEEDS Consortium. A call for researchers to join the duction varies widely: a scoping review. J Clin Epidemiol 2021;139:
META-MICROBLEEDS Consortium. Lancet Neurol 2016;15:900. 287e96.
[12] Furlan AD, Malmivaara A, Chou R, Maher CG, Deyo RA, [17] International Committee of Medical Journal Editors. Recommenda-
Schoene M, et al. 2015 updated method guideline for systematic re- tions for the conduct, reporting, editing and publication of scholarly
views in the Cochrane back and neck group. Spine 2015;40: work in medical journals. Available at http://www.ICMJE.org. Ac-
1660e73. cessed September 20, 2022.