Research Methods and Reporting

BMJ: first published as 10.1136/bmj.i4919 on 12 October 2016. Downloaded from http://www.bmj.com/ on 25 February 2020 at India:BMJ-PG Sponsored. Protected by copyright.
open access
ROBINS-I: a tool for assessing risk of bias in non-randomised
studies of interventions
Jonathan AC Sterne,1 Miguel A Hernán,2 Barnaby C Reeves,3 Jelena Savović,1,4 Nancy D Berkman,5
Meera Viswanathan,6 David Henry,7 Douglas G Altman,8 Mohammed T Ansari,9 Isabelle Boutron,10
James R Carpenter,11 An-Wen Chan,12 Rachel Churchill,13 Jonathan J Deeks,14 Asbjørn Hróbjartsson,15
Jamie Kirkham,16 Peter Jüni,17 Yoon K Loke,18 Theresa D Pigott,19 Craig R Ramsay,20 Deborah Regidor,21
Hannah R Rothstein,22 Lakhbir Sandhu,23 Pasqualina L Santaguida,24 Holger J Schünemann,25
Beverly Shea,26 Ian Shrier,27 Peter Tugwell,28 Lucy Turner,29 Jeffrey C Valentine,30 Hugh Waddington,31
Elizabeth Waters,32 George A Wells,33 Penny F Whiting,34 Julian PT Higgins35

For numbered affiliations see Non-randomised studies of the such as cohort studies and case-control studies in
end of article. which intervention groups are allocated during the
Correspondence to: J A C Sterne effects of interventions are critical to course of usual treatment decisions, and quasi-ran-
[email protected] many areas of healthcare evaluation, domised studies in which the method of allocation
Additional material is published
online only. To view please visit but their results may be biased. It is falls short of full randomisation. Non-randomised
studies can provide evidence additional to that avail-
the journal online.
therefore important to understand
Cite this as: BMJ 2016;355:i4919 able from randomised trials about long term out-
http://dx.doi.org/10.1136/bmj.i4919 and appraise their strengths and comes, rare events, adverse effects and populations
weaknesses. We developed ROBINS-I that are typical of real world practice.1 2  The availabil-
(“Risk Of Bias In Non-randomised ity of linked databases and compilations of electronic
health records has enabled NRSI to be conducted in
Studies - of Interventions”), a new large representative population cohorts.3 For many
tool for evaluating risk of bias in types of organisational or public health interventions,
estimates of the comparative NRSI are the main source of evidence about the likely
impact of the intervention because randomised trials
effectiveness (harm or benefit) of are difficult or impossible to conduct on an area-wide
interventions from studies that did basis. Therefore systematic reviews addressing the
not use randomisation to allocate effects of health related interventions often include
NRSI. It is essential that methods are available to eval-
units (individuals or clusters of uate these studies, so that clinical, policy, and individ-
individuals) to comparison groups. ual decisions are transparent and based on a full
The tool will be particularly useful to understanding of the strengths and weaknesses of the
evidence.
those undertaking systematic Many tools to assess the methodological quality of
reviews that include non-randomised observational studies in the context of a systematic
studies. review have been proposed.4 5  The Newcastle-Ottawa6 
and Downs-Black7  tools have been two of the most pop-
Non-randomised studies of the effects of interventions ular: both were on a shortlist of methodologically
(NRSI) are critical to many areas of healthcare evalua- sound tools,5  but each includes items relating to exter-
tion. Designs of NRSI that can be used to evaluate the nal as well as internal validity and a lack of comprehen-
effects of interventions include observational studies sive manuals means that instructions may be
interpreted differently by different users.5
In the past decade, major developments have been
Summary points made in tools to assess study validity. A shift in focus
•  Non-randomised studies of the effects of interventions are critical to many areas from methodological quality to risk of bias has been
of healthcare evaluation but are subject to confounding and a range of other accompanied by a move from checklists and numeric
potential biases scores towards domain-based assessments in which
•  We developed, piloted, and refined a new tool, ROBINS-I, to assess “Risk Of Bias different types of bias are considered in turn. Examples
In Non-randomised Studies - of Interventions” are the Cochrane Risk of Bias tool for randomised
•  The tool views each study as an attempt to emulate (mimic) a hypothetical ­trials,8  the QUADAS 2 tool for diagnostic test accuracy
pragmatic randomised trial, and covers seven distinct domains through which studies,9  and the ROBIS tool for systematic reviews.10 
bias might be introduced However, there is no satisfactory domain-based assess-
ment tool for NRSI.4
•  We use “signalling questions” to help users of ROBINS-I to judge risk of bias
In this paper we describe the development of
within each domain
ROBINS-I (“Risk Of Bias In Non-randomised
•  The judgements within each domain carry forward to an overall risk of bias
­S tudies  - of Interventions”), which is concerned
judgement across bias domains for the outcome being assessed
with evaluating risk of bias in estimates of the

the bmj | BMJ 2016;355:i4919 | doi: 10.1136/bmj.i4919 1

bmj.i4919.indd 1 11-10-2016 17:40:06

 Research Methods and Reporting

BMJ: first published as 10.1136/bmj.i4919 on 12 October 2016. Downloaded from http://www.bmj.com/ on 25 February 2020 at India:BMJ-PG Sponsored. Protected by copyright.
e­ ffectiveness or safety (benefit or harm) of an inter- need not be feasible or ethical: for example, it could
vention from studies that did not use randomisation compare individuals who were and were not assigned
to allocate interventions. to start smoking. Description of the target trial for the
NRSI being assessed includes details of the popula-
Development of a new tool tion, experimental intervention, comparator, and out-
We developed the tool over three years, largely by comes of interest. Correspondingly, we define bias as a
expert consensus, and following the seven principles systematic difference between the results of the NRSI
we previously described for assessing risk of bias in and the results expected from the target trial. Such
clinical trials.8  A core group coordinated develop- bias is distinct from issues of generalisability (applica-
ment of the tool, including recruitment of collabora- bility or transportability) to types of individuals who
tors, preparation and revision of documents, and were not included in the study.
administrative support. An initial scoping meeting in
October 2011 was followed by a survey of Cochrane The effect of interest
Review Groups in March 2012 to gather information In the target trial, the effect of interest will typically be
about the methods they were using to assess risk of that of either:
bias in NRSI. A meeting in April 2012 identified the
relevant bias domains and established working 1. Assignment to intervention at baseline (start of follow
groups focusing on each of these. We agreed at this up), regardless of the extent to which the intervention
stage to use the approach previously adopted in the was received during the follow-up (sometimes
QUADAS-2 tool, in which answers to “signalling ques- referred to as the “intention-to-treat” effect)
tions” help reviewers judge the risk of bias within 2. Starting and adhering to the intervention as indi-
each domain.9 We distributed briefing documents to cated in the trial protocol (sometimes referred to as
working groups in June 2012, specifying consider- the “per-protocol” effect).
ations for how signalling questions should be formu-
lated and how answers to these would lead to a risk of For example, in a trial of cancer screening, our interest
bias judgement. We also identified methodological might be in the effect of either sending an invitation to
issues that would underpin the new tool: these are attend screening or of responding to the invitation and
described below. undergoing screening.
After collation and harmonisation by the core group Analogues of these effects can be defined for NRSI.
of the working groups’ contributions, all collaborators For example, the intention-to-treat effect in a study
considered draft signalling questions and agreed on the comparing aspirin with no aspirin can be approximated
main features of the new tool during a two-day face-to- by the effect of being prescribed aspirin or (if using dis-
face meeting in March 2013. A preliminary version of pensing rather than prescription data) the effect of
the tool was piloted within the working groups between starting aspirin (this corresponds to the intention-to-
September 2013 and March 2014, using NRSI in several treat effect in a trial in which participants assigned to
review topic areas. Substantial revisions, based on an intervention always start that intervention). Alterna-
results of the piloting, were agreed by leads of working tively, we might be interested in the effect of starting
groups in June 2014. Further piloting took place, along and adhering to aspirin.
with a series of telephone interviews with people using The type of effect of interest influences assessments
the tool for the first time that explored whether they of risk of bias related to deviations from intervention.
were interpreting the tool and the guidance as intended. When the effect of interest is that of assignment to (or
We posted version 1.0.0, along with detailed guidance, starting) intervention, risk of bias assessments gener-
at www.riskofbias.info in September 2014. We ally need not be concerned with post-baseline devia-
explained the tool during a three-day workshop involv- tions from interventions.13 By contrast, unbiased
ing members of Cochrane Review Groups in December estimation of the effect of starting and adhering to inter-
2014, and applied it in small groups to six papers report- vention requires consideration of both adherence and
ing NRSI. Further modifications to the tool, particularly differences in additional interventions (“co-interven-
regarding wording, were based on feedback from this tions”) between intervention groups.
event and from subsequent training events conducted
during 2015. Domains of bias
We achieved consensus on seven domains through
Methodological issues in assessing risk of bias in which bias might be introduced into a NRSI (see table 1
non-randomised studies and appendix in supplementary data). The first two
The target trial domains, covering confounding and selection of partici-
Evaluations of risk of bias in the results of NRSI are pants into the study, address issues before the start of the
facilitated by considering each NRSI as an attempt to interventions that are to be compared (“baseline”). The
emulate (mimic) a “target” trial. This is the hypotheti- third domain addresses classification of the interven-
cal pragmatic randomised trial, conducted on the tions themselves. The other four domains address issues
same participant group and without features putting it after the start of interventions: biases due to deviations
at risk of bias, whose results would answer the ques- from intended interventions, missing data, measure-
tion addressed by the NRSI.11 12 Such a “target” trial ment of outcomes, and selection of the reported result.

2 doi: 10.1136/bmj.i4919 | BMJ 2016;355:i4919 | the bmj

bmj.i4919.indd 2 11-10-2016 17:40:06

 Research Methods and Reporting

BMJ: first published as 10.1136/bmj.i4919 on 12 October 2016. Downloaded from http://www.bmj.com/ on 25 February 2020 at India:BMJ-PG Sponsored. Protected by copyright.
Table 1 | Bias domains included in ROBINS-I
Domain Explanation
Pre-intervention Risk of bias assessment is mainly distinct from assessments of randomised trials
Bias due to Baseline confounding occurs when one or more prognostic variables (factors that predict the outcome of interest) also predicts the intervention
confounding received at baseline
ROBINS-I can also address time-varying confounding, which occurs when individuals switch between the interventions being compared and when
post-baseline prognostic factors affect the intervention received after baseline
Bias in selection of When exclusion of some eligible participants, or the initial follow-up time of some participants, or some outcome events is related to both
participants into the intervention and outcome, there will be an association between interventions and outcome even if the effects of the interventions are identical
study This form of selection bias is distinct from confounding—A specific example is bias due to the inclusion of prevalent users, rather than new users, of
an intervention
At intervention Risk of bias assessment is mainly distinct from assessments of randomised trials
Bias in classification of Bias introduced by either differential or non-differential misclassification of intervention status
interventions Non-differential misclassification is unrelated to the outcome and will usually bias the estimated effect of intervention towards the null
Differential misclassification occurs when misclassification of intervention status is related to the outcome or the risk of the outcome, and is likely to
lead to bias
Post-intervention Risk of bias assessment has substantial overlap with assessments of randomised trials
Bias due to deviations Bias that arises when there are systematic differences between experimental intervention and comparator groups in the care provided, which
from intended represent a deviation from the intended intervention(s)
interventions Assessment of bias in this domain will depend on the type of effect of interest (either the effect of assignment to intervention or the effect of starting
and adhering to intervention).
Bias due to missing Bias that arises when later follow-up is missing for individuals initially included and followed (such as differential loss to follow-up that is affected by
data prognostic factors); bias due to exclusion of individuals with missing information about intervention status or other variables such as confounders
Bias in measurement of Bias introduced by either differential or non-differential errors in measurement of outcome data. Such bias can arise when outcome assessors are
outcomes aware of intervention status, if different methods are used to assess outcomes in different intervention groups, or if measurement errors are related
to intervention status or effects
Bias in selection of the Selective reporting of results in a way that depends on the findings and prevents the estimate from being included in a meta-analysis (or other
reported result synthesis)

For the first three domains, risk of bias assessments care,” or an alternative intervention. It is important
for NRSI are mainly distinct from assessments of ran- to  consider in advance the confounding factors and
domised trials because randomisation, if properly co-interventions that have the potential to lead to bias.
implemented, protects against biases that arise before Relevant confounding domains are the prognostic fac-
the start of intervention. However, randomisation does tors that predict whether an individual receives one or
not protect against biases that arise after the start of the other intervention of interest. Relevant co-interven-
intervention. Therefore, there is substantial overlap for tions are those that individuals might receive with or
the last four domains between bias assessments in after starting the intervention of interest and that are
NRSI and randomised trials. both related to the intervention received and prognostic
Variation in terminology proved a challenge to devel- for the outcome of interest. Both confounding domains
opment of ROBINS-I. The same terms are sometimes and co-interventions are likely to be identified through
used to refer to different types of bias in randomised the expert knowledge of members of the review group
trials and NRSI literature,13  and different types of bias and through initial (scoping) reviews of the literature.
are often described by a host of different terms: those Discussions with health professionals who make
used within ROBINS-I are shown in the first column of ­intervention decisions for the target patient or popula-
table 1. tion groups may also help in identification of prognos-
tic factors that influence treatment decisions.
The risk of bias tool, ROBINS-I
The full ROBINS-I tool is shown in tables A, B, and C in Assessing a specific study
the supplementary data. The assessment of each NRSI included in the review
involves following the six steps below (supplementary
Planning the risk of bias assessment table A, stage II). Steps 3 to 6 should be repeated for
It is very important that experts in both subject matter each key outcome of interest:
and epidemiological methods are included in any team
evaluating a NRSI. The risk of bias assessment should 1. Specify the research question through consideration
begin with consideration of what problems might arise, of a target trial
in the context of the research question, in making a 2. Specify the outcome and result being assessed
causal assessment of the effect of the intervention(s) of 3. For the specified result, examine how the confound-
interest on the basis of NRSI. This will be based on ers and co-interventions were addressed
experts’ knowledge of the literature: the team should 4. Answer signalling questions for the seven bias
also address whether conflicts of interest might affect domains
experts’ judgements. 5. Formulate risk of bias judgements for each of the
The research question is conceptualised by defining seven bias domains, informed by answers to the sig-
the population, experimental intervention, comparator, nalling questions
and outcomes of interest (supplementary table A, stage 6. Formulate an overall judgement on risk of bias for the
I). The comparator could be “no intervention,” “usual outcome and result being assessed.

the bmj | BMJ 2016;355:i4919 | doi: 10.1136/bmj.i4919 3

bmj.i4919.indd 3 11-10-2016 17:40:06

 Research Methods and Reporting

BMJ: first published as 10.1136/bmj.i4919 on 12 October 2016. Downloaded from http://www.bmj.com/ on 25 February 2020 at India:BMJ-PG Sponsored. Protected by copyright.
Examination of confounders and co-interventions is, concerns should be expressed only about issues that
involves determining whether the important confound- are likely to affect the ability to draw valid conclusions
ers and co-interventions were measured or adminis- from the study: a serious risk of a very small degree of
tered in the study at hand, and whether additional bias should not be considered “Serious risk” of bias. The
confounders and co-interventions were identified. Sup- “No information” category should be used only when
plementary table A provides a structured approach to insufficient data are reported to permit a judgement.
assessing the potential for bias due to confounding and The judgements within each domain carry forward to
co-interventions and includes the full tool with the sig- an overall risk of bias judgement for the outcome being
nalling questions to be addressed within each bias assessed (across bias domains, that is), as summarised
domain. in table 2 (also saved as supplementary table D). The
The signalling questions are broadly factual in nature key to applying the tool is to make domain-level judge-
and aim to facilitate judgements about the risk of bias. ments about risk of bias that have the same meaning
The response options are: “Yes”; “Probably yes”; across domains with respect to concern about the
“­Probably no”; “No”; and “No information”. Some ques- impact of bias on the trustworthiness of the result. If
tions are answered only if the response to a previous domain-level judgements are made consistently, then
question is “Yes” or “Probably yes” (or “No” or “Proba- judging the overall risk of bias for a particular outcome
bly no”). Responses of “Yes” are intended to have similar is relatively straightforward. For instance, a “Serious
implications to responses of “Probably yes” (and simi- risk” of bias in one domain means the effect estimate
larly for “No” and “Probably no”), but allow for a dis- from the study is at serious risk of bias or worse, even if
tinction between something that is known and the risk of bias is judged to be lower in the other
something that is likely to be the case. Free text should domains.
be used to provide support for each answer, using direct It would be highly desirable to know the magnitude
quotations from the text of the study where possible. and direction of any potential biases identified, but this
Responses to signalling questions provide the basis is considerably more challenging than judging the risk
for domain-level judgements about risk of bias, which of bias. The tool includes an optional component to pre-
then provide the basis for an overall risk of bias judge- dict the direction of the bias for each domain, and over-
ment for a particular outcome. The use of the word all. For some domains, the bias is most easily thought of
“judgement” to describe this process is important and as being towards or away from the null. For example,
reflects the need for review authors to consider both the suspicion of selective non-reporting of statistically
severity of the bias in a particular domain and the rela- non-significant results would suggest bias against the
tive consequences of bias in different domains. null. However, for other domains (in particular con-
The categories for risk of bias judgements are “Low founding, selection bias and forms of measurement
risk”, “Moderate risk”, “Serious risk” and “Critical risk” bias such as differential misclassification), the bias
of bias. Importantly, “Low risk” corresponds to the risk needs to be thought of as an increase or decrease in the
of bias in a high quality randomised trial. Only excep- effect estimate and not in relation to the null. For exam-
tionally will an NRSI be assessed as at low risk of bias ple, confounding bias that decreases the effect estimate
due to confounding. Criteria for reaching risk of bias would be towards the null if the true risk ratio were
judgements for the seven domains are provided in sup- greater than 1, and away from the null if the risk ratio
plementary tables B and C. If none of the answers to the were less than 1.
signalling questions for a domain suggests a potential
problem then risk of bias for the domain can be judged Discussion
to be low. Otherwise, potential for bias exists. Review We developed a tool for assessing risk of bias in the
authors must then make a judgement on the extent to results of non-randomised studies of interventions
which the results of the study are at risk of bias. “Risk of that  addresses weaknesses in previously available
bias” is to be interpreted as “risk of material bias”. That approaches. 4  Our approach builds on recent

Table 2 | Interpretation of domain-level and overall risk of bias judgements in ROBINS-I*

Judgement Within each domain Across domains Criterion
Low risk of The study is comparable to a well performed The study is comparable to a well performed The study is judged to be at low risk of bias
bias randomised trial with regard to this domain randomised trial for all domains
Moderate The study is sound for a non-randomised study with The study provides sound evidence for a non- The study is judged to be at low or moderate
risk of bias regard to this domain but cannot be considered randomised study but cannot be considered risk of bias for all domains
comparable to a well performed randomised trial comparable to a well performed randomised trial
Serious risk The study has some important problems in this The study has some important problems The study is judged to be at serious risk of
of bias domain bias in at least one domain, but not at critical
risk of bias in any domain
Critical risk The study is too problematic in this domain to provide The study is too problematic to provide any useful The study is judged to be at critical risk of
of bias any useful evidence on the effects of intervention evidence and should not be included in any synthesis bias in at least one domain
No No information on which to base a judgement about No information on which to base a judgement about There is no clear indication that the study is at
information risk of bias for this domain risk of bias serious or critical risk of bias and there is a
lack of information in one or more key domains
of bias (a judgement is required for this)
*Also saved as supplementary table D.

4 doi: 10.1136/bmj.i4919 | BMJ 2016;355:i4919 | the bmj

bmj.i4919.indd 4 11-10-2016 17:40:06

 Research Methods and Reporting

BMJ: first published as 10.1136/bmj.i4919 on 12 October 2016. Downloaded from http://www.bmj.com/ on 25 February 2020 at India:BMJ-PG Sponsored. Protected by copyright.
­ evelopments in risk of bias assessment of randomised
d We developed the ROBINS-I tool primarily for use in
trials and diagnostic test accuracy studies.8 9 Key fea- the context of a systematic review. Broader potential
tures of ROBINS-I include specification of the target uses include the assessment of funding applications
trial and effect of interest, use of signalling questions to and peer review of journal submissions. Furthermore,
inform judgements of risk of bias, and assessments ROBINS-I may be used to guide researchers about issues
within seven bias domains. to consider when designing a primary study to evaluate
The ROBINS-I tool was developed through consensus the effect(s) of an intervention.
among a group that included both methodological Figure 1  summarises the process of assessing risk of
experts and systematic review authors and editors, and bias using the tool in the context of a systematic review
was substantially revised based on extensive piloting of NRSI. To draw conclusions about the extent to which
and user feedback. It includes a structured approach to observed intervention effects might be causal, the stud-
assessment of risk of bias due to confounding that ies included in the review should be compared and con-
starts at the review protocol stage. Use of ROBINS-I trasted so that their strengths and weaknesses can be
requires that review groups include members with sub- considered jointly. Studies with different designs may
stantial methodological expertise and familiarity with present different types of bias, and “triangulation” of
modern epidemiological thinking. We tried to make findings across these studies may provide assurance
ROBINS-I as accessible and easy to use as possible, either that the biases are minimal or that they are real.
given the requirement for comprehensive risk of bias Syntheses of findings across studies through meta-anal-
assessments that are applicable to a wide range of study ysis must consider the risks of bias in the studies avail-
designs and analyses. An illustrative assessment using able. We recommend against including studies assessed
ROBINS-I can be found at www.riskofbias.info; detailed as at “Critical risk” of bias in any meta-analysis, and
guidance and further training materials will also be advocate caution for studies assessed as at “Serious
available. risk” of bias. Subgroup analyses (in which intervention
ROBINS-I separates relatively factual answers to sig- effects are estimated separately according to risk of
nalling questions from more subjective judgements bias), meta-regression analyses, and sensitivity analy-
about risk of bias. We hope that the explicit links ses (excluding studies at higher risk of bias) might be
between answers to signalling questions and risk of considered, either within specific bias domains or over-
bias judgements will improve reliability of the all. Risk of bias assessments might alternatively be used
domain-specific and overall risk of bias assessments.14 as the basis for deriving adjustments for bias through
Nonetheless, we expect that the technical difficulty in prior distributions in Bayesian meta-analyses.15 16
making risk of bias judgements will limit reliability. The GRADE system for assessing confidence in esti-
Despite this, ROBINS-I provides a comprehensive and mates of the effects of interventions currently assigns a
structured approach to assessing non-randomised starting rating of “Low certainty, confidence or quality”
studies of interventions. It should therefore facilitate to non-randomised studies, a downgrading by default
debates and improve mutual understanding about the of two levels.17 ROBINS-I provides a thorough assess-
ways in which bias can influence effects estimated in ment of risk of bias in relation to a hypothetical ran-
NRSI, and clarify reasons for disagreements about spe- domised trial, and “Low risk” of bias corresponds to the
cific risk of bias judgements. Note that the tool focuses risk of bias in a high quality randomised trial. This
specifically on bias and does not address problems opens up the possibility of using the risk of bias assess-
related to imprecision of results, for example when sta- ment, rather than the lack of randomisation per se, to
tistical analyses fail to account for clustering or match- determine the degree of downgrading of a study result,
ing of participants. and means that results of NRSI and randomised trials
could be synthesised if they are assessed to be at similar
risks of bias. In general, however, we advocate analys-
Stage I: Planning ing these study designs separately and focusing on evi-
Specify research question; list potential confounding domains and co-interventions
dence from NRSI when evidence from trials is not
For each outcome available.
For each study Planned developments of ROBINS-I include further
Stage II: Risk of bias assessment for specific result consideration of the extent to which it works for specific
types of NRSI, such as self-controlled designs, con-
Stage II-1: Stage II-2: Stage II-3:
Specify target trial Select the Examine confounders trolled before-and-after studies, interrupted time series
and effect of interest result to assess and co-interventions studies, and studies based on regression discontinuity
Stage II-4: Stage II-5: Stage II-6: and instrumental variable analyses. We also plan to
Answer signalling Risk of bias judgment Overall risk of bias develop interactive software to facilitate use of ROB-
questions for each domain judgment for the result
INS-I. Furthermore, the discussions that led up to the
tool will inform a reconsideration of the tool for ran-
Stage III: Overall risk of bias assessment
‘Triangulate’ across studies
domised trials, particularly in the four post-interven-
tion domains.8
The role of NRSI in informing treatment decisions
Fig 1 | Summary of the process of assessing risk of bias in a systematic review of remains controversial. Because randomised trials are
non-randomised studies of interventions (NRSI) expensive, time consuming, and may not reflect real

the bmj | BMJ 2016;355:i4919 | doi: 10.1136/bmj.i4919 5

bmj.i4919.indd 5 11-10-2016 17:40:06

 Research Methods and Reporting

BMJ: first published as 10.1136/bmj.i4919 on 12 October 2016. Downloaded from http://www.bmj.com/ on 25 February 2020 at India:BMJ-PG Sponsored. Protected by copyright.
23Division of General Surgery, University of Toronto, Toronto, Canada
world experience with healthcare interventions,
24Department of Clinical Epidemiology and Biostatistics, McMaster
research funders are enthusiastic about the possible
University, Hamilton, Ontario, L8S 4K1, Canada
use of observational studies to provide evidence about 25Departments of Clinical Epidemiology and Biostatistics and of
the comparative effectiveness of different interven- Medicine, Cochrane Applicability and Recommendations Methods
tions,18  and encourage use of large, routinely collected (GRADEing) Group, MacGRADE center, Ontario, L8N 4K1, Canada
datasets assembled through data linkage.18  However, 26Ottawa Hospital Research Institute, Center for Practice Changing

fear that evidence from NRSI may be biased, based on Research and School of Epidemiology, Public Health and Preventive
Medicine, Faculty of Medicine, University of Ottawa, Ottawa,
misleading results of some NRSI,19 20  has led to caution Ontario, K1H 8M5, Canada
in their use in making judgements about efficacy. There 27Centre for Clinical Epidemiology, Lady Davis Institute, Jewish

is greater confidence in the capacity of NRSI to quantify General Hospital, McGill University, Montreal, Quebec, Canada
28Department of Medicine and School of Epidemiology, Public
uncommon adverse effects of interventions.21  We
Health and Preventive Medicine, University of Ottawa, Ottawa,
believe that evidence from NRSI should complement Ontario, Canada
that from randomised trials, such as in providing evi- 29Ottawa Hospital Research Institute, Ottawa, ON, Canada

dence about effects on rare and adverse outcomes and 30University of Louisville, Louisville, KY 40292, USA

long term effects to be balanced against the outcomes 31International Initiative for Impact Evaluation, London School of

more readily addressed in randomised trials.22 Hygiene and Tropical Medicine, and London International
Development Centre, London, UK
32Jack Brockhoff Child Health & Wellbeing Program, Melbourne

Author affiliations School of Population and Global Health, University of Melbourne,

1School Melbourne, VIC 3010, Australia
of Social and Community Medicine, University of Bristol,
33School of Epidemiology, Public Health and Preventive Medicine
Bristol BS8 2PS, UK
2Department of Epidemiology, Harvard T.H. Chan School of Public and Director, Cardiovascular Research Methods Centre, University of
Ottawa Heart Institute, Ottawa, Ontario, K1Y 4W7, Canada
Health, Boston, Massachusetts, USA; and Department of
34School of Social and Community Medicine, University of Bristol,
Biostatistics, Harvard T.H. Chan School of Public Health, Boston,
Massachusetts, USA; and Harvard-Massachusetts Institute of Bristol BS8 2PS, UK; and National Institute for Health Research
Technology Division of Health Sciences and Technology, Boston, Collaboration for Leadership in Applied Health Research and Care
Massachusetts, USA West (NIHR CLAHRC West) at University Hospitals Bristol NHS
3School of Clinical Sciences, University of Bristol, Bristol, BS2 8HW, UK Foundation Trust, Bristol BS1 2NT, UK
35School of Social and Community Medicine, University of Bristol,
4National Institute for Health Research Collaboration for Leadership in
Bristol BS8 2PS, UK
Applied Health Research and Care West (NIHR CLAHRC West) at
University Hospitals Bristol NHS Foundation Trust, Bristol BS1 2NT, UK The ROBINS-I tool is reproduced from riskofbias.info with the
5Program on Health Care Quality and Outcomes, Division of Health permission of the authors. The tool should not be modified for use.
Services and Social Policy Research, RTI International, Research We thank Professor Jan Vandenbroucke for his contributions to
Triangle Park, NC 27709, USA discussions within working groups and during face-to-face meetings.
6RTI-UNC Evidence-based Practice Center, RTI International, Professor David Moher and Dr Vivan Welch contributed to the grant
application and to initial discussions. We thank all those individuals
Research Triangle Park, NC 27709, USA
who contributed to development of ROBINS-I through discussions
7Dalla Lana School of Public Health, University of Toronto, Toronto,
during workshops and training events, and through their work on
Ontario, Canada piloting.
8Centre for Statistics in Medicine, Nuffield Department of
Contributions of authors: JACS, BCR, JS, LT, YKL, EW, CRR, PT, GAW,
Orthopaedics, Rheumatology & Musculoskeletal Sciences, and JPTH conceived the project. JACS, JPTH, BCR, JS, and LT oversaw
University of Oxford, Oxford OX3 7LD, UK the project. JACS, DH, JPTH, IB, and BRR led working groups. MAH
9School of Epidemiology, Public Health and Preventive Medicine, Faculty developed the idea of the target trial and its role in ROBINS-I. NDB and
of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada MV undertook cognitive testing of previous drafts of the tool. All
10METHODS Team, Centre of Epidemiology and Statistics Sorbonne authors contributed to development of ROBINS-I and to writing
Paris Cité Research, INSERM UMR 1153, University Paris Descartes, associated guidance. JACS, MAH, and JPTH led on drafting the
Paris, France manuscript. All authors reviewed and commented on drafts of the
11Department of Medical Statistics, London School of Hygiene and
manuscript.
Tropical Medicine and MRC Clinical Trials Unit at UCL, London, UK Funding: Development of ROBINS-I was funded by a Methods
12Women’s College Research Institute, Department of Medicine,
Innovation Fund grant from Cochrane and Medical Research Council
(MRC) grant MR/M025209/1. Sterne and Higgins are members of the
University of Toronto, Canada MRC Integrative Epidemiology Unit at the University of Bristol, which is
13Centre for Reviews and Dissemination, University of York, York, YO10
supported by the MRC and the University of Bristol (grant MC_
5DD, UK UU_12013/9). This research was partly funded by NIH grant P01
14Institute of Applied Health Research, University of Birmingham, CA134294. Sterne was supported by National Institute for Health
Birmingham B15 2TT, UK Research (NIHR) Senior Investigator award NF-SI-0611-10168. Savović
15Center for Evidence-Based Medicine, University of Southern and Whiting were supported by NIHR Collaboration for Leadership in
Applied Health Research and Care West (NIHR CLAHRC West). Reeves
Denmark & Odense University Hospital, 5000 Odense C, Denmark
was supported by the NIHR Bristol Biomedical Research Unit in
16Department of Biostatistics, University of Liverpool, Liverpool, L69
Cardiovascular Disease. None of the funders had a role in the
3GL, UK development of the ROBINS-I tool, although employees of Cochrane
17Applied Health Research Centre (AHRC), Li Ka Shing Knowledge contributed to some of the meetings and workshops. The views
Institute of St Michael’s Hospital, and Department of Medicine, expressed are those of the authors and not necessarily those of
University of Toronto, Toronto, Ontario, Canada Cochrane, the NHS, NIHR, or Department of Health.
18Norwich Medical School, University of East Anglia, Norwich NR4 7TJ, UK Competing interests: All authors have completed the ICMJE uniform
19School of Education, Loyola University Chicago, Chicago, IL 60611, USA disclosure form at http://www.icmje.org/coi_disclosure.pdf and
20Health Services Research Unit, University of Aberdeen, Aberdeen,
declare: grants from Cochrane, MRC, and NIHR during the conduct of
the study. Dr Carpenter reports personal fees from Pfizer, grants and
AB25 2ZD, UK. non-financial support from GSK and grants from Novartis, outside the
21Evidence Services, Kaiser Permanente, Care Management Institute,
submitted work. Dr Reeves is a co-convenor of the Cochrane
Oakland, CA 94612, USA Non-Randomised Studies Methods Group. The authors report no other
22Department of Management, Zicklin School of Business, Baruch relationships or activities that could appear to have influenced the
College—CUNY, New York, NY 10010, USA submitted work.

6 doi: 10.1136/bmj.i4919 | BMJ 2016;355:i4919 | the bmj

bmj.i4919.indd 6 11-10-2016 17:40:06

 Research Methods and Reporting

BMJ: first published as 10.1136/bmj.i4919 on 12 October 2016. Downloaded from http://www.bmj.com/ on 25 February 2020 at India:BMJ-PG Sponsored. Protected by copyright.
Provenance: The authors are epidemiologists, statisticians, 12 Hernán MA, Robins JM. Using big data to emulate a target trial when a
systematic reviewers, trialists, and health services researchers, many randomized trial is not available. Am J Epidemiol 2016;183:758-64.
of whom are involved with Cochrane systematic reviews, methods doi:10.1093/aje/kwv254.
groups and training events. Development of ROBINS-I was informed 13 Mansournia MA, Higgins JPT, Sterne JAC, et al. Biases in randomized
by relevant methodological literature, previously published tools for trials: a conversation between trialists and epidemiologists.
assessing methodological quality of non-randomised studies, Epidemiology [forthcoming].
systematic reviews of such tools and relevant literature, and by the 14 Hartling L, Hamm MP, Milne A, et al. Testing the risk of bias tool
showed low reliability between individual reviewers and across
authors’ experience of developing tools to assess risk of bias in
consensus assessments of reviewer pairs. J Clin Epidemiol
randomised trials, diagnostic test accuracy studies, and systematic
2013;66:973-81. doi:10.1016/j.jclinepi.2012.07.005.
reviews. All authors contributed to development of ROBINS-I and to 15 Turner RM, Spiegelhalter DJ, Smith GCS, Thompson SG. Bias modelling
writing associated guidance. All authors reviewed and commented in evidence synthesis. J R Stat Soc Ser A Stat Soc 2009;172:21-47.
on drafts of the manuscript. J Sterne will act as guarantor. doi:10.1111/j.1467-985X.2008.00547.x.
This is an Open Access article distributed in accordance with the 16 Wilks DC, Sharp SJ, Ekelund U, et al. Objectively measured physical
Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, activity and fat mass in children: a bias-adjusted meta-analysis of
which permits others to distribute, remix, adapt, build upon this work prospective studies. PLoS One 2011;6:e17205. doi:10.1371/journal.
non-commercially, and license their derivative works on different terms, pone.0017205.
17 Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A.
provided the original work is properly cited and the use is non-
GRADE guidelines: a new series of articles in the Journal of Clinical
commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.
Epidemiology. J Clin Epidemiol 2011;64:380-2. doi:10.1016/j.
1 Black N. Why we need observational studies to evaluate the jclinepi.2010.09.011.
effectiveness of health care. BMJ 1996;312:1215-8. doi:10.1136/ 18 Sox HC, Greenfield S. Comparative effectiveness research: a report
bmj.312.7040.1215. from the Institute of Medicine. Ann Intern Med 2009;151:203-5.
2 Feinstein AR. An additional basic science for clinical medicine: II. The doi:10.7326/0003-4819-151-3-200908040-00125.
limitations of randomized trials. Ann Intern Med 1983;99:544-50. 19 Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C. Mortality in
doi:10.7326/0003-4819-99-4-544. randomized trials of antioxidant supplements for primary and
3 Strom B. Overview of automated databases in secondary prevention: systematic review and meta-analysis. JAMA
pharmacoepidemiology. In: Strom BL, Hennessy S, eds. 2007;297:842-57. doi:10.1001/jama.297.8.842.
Pharmacoepidemiology. 5th ed. Wiley, 2012. 20 Hernán MA, Alonso A, Logan R, et al. Observational studies analyzed
4 Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and like randomized experiments: an application to postmenopausal
susceptibility to bias in observational studies in epidemiology: a hormone therapy and coronary heart disease. Epidemiology
systematic review and annotated bibliography. Int J Epidemiol 2008;19:766-79. doi:10.1097/EDE.0b013e3181875e61.
2007;36:666-76. doi:10.1093/ije/dym018. 21 Golder S, Loke YK, Bland M. Meta-analyses of adverse effects data
5 Deeks JJ, Dinnes J, D’Amico R, et al. International Stroke Trial derived from randomised controlled trials as compared to
Collaborative Group European Carotid Surgery Trial Collaborative observational studies: methodological overview. PLoS Med
Group. Evaluating non-randomised intervention studies. Health 2011;8:e1001026. doi:10.1371/journal.pmed.1001026.
Technol Assess 2003;7:iii-x, 1-173. doi:10.3310/hta7270. 22 Schünemann HJ, Tugwell P, Reeves BC, et al. Non-randomized studies as
6 Wells GA, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale a source of complementary, sequential or replacement evidence for
(NOS) for assessing the quality of nonrandomised studies in randomized controlled trials in systematic reviews on the effects of
meta-analyses. 2008. http://www.ohri.ca/programs/clinical_ interventions. Res Synth Methods 2013;4:49-62. doi:10.1002/jrsm.1078.
epidemiology/oxford.asp (accessed 1/03/2016).
7 Downs SH, Black N. The feasibility of creating a checklist for the
assessment of the methodological quality both of randomised and © BMJ Publishing Group Ltd 2016
non-randomised studies of health care interventions. J Epidemiol
Community Health 1998;52:377-84. doi:10.1136/jech.52.6.377. Appendix: The seven domains of bias addressed in
8 Higgins JPT, Altman DG, Gøtzsche PC, et al. Cochrane Bias Methods
Group Cochrane Statistical Methods Group. The Cochrane the ROBINS-I assessment tool
Collaboration’s tool for assessing risk of bias in randomised trials. BMJ Supplementary tables. Table A: Risk Of Bias In
2011;343:d5928. doi:10.1136/bmj.d5928.
9 Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2 Group.
Non-randomized Studies – of Interventions
QUADAS-2: a revised tool for the quality assessment of diagnostic (ROBINS-I) assessment tool. Table B: Reaching risk
accuracy studies. Ann Intern Med 2011;155:529-36. of bias judgements in ROBINS-I: pre-intervention
doi:10.7326/0003-4819-155-8-201110180-00009.
10 Whiting P, Savović J, Higgins JPT, et al. ROBIS group. ROBIS: A new tool and at-intervention domains. Table C: Reaching risk
to assess risk of bias in systematic reviews was developed. J Clin of bias judgements in ROBINS-I: post-intervention
Epidemiol 2016;69:225-34. doi:10.1016/j.jclinepi.2015.06.005.
11 Institute of Medicine. Ethical and Scientific Issues in Studying the
domains. Table D: Interpretation of domain-level
Safety of Approved Drugs.The National Academies Press, 2012. and overall risk of bias judgements in ROBINS-I

No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe

bmj.i4919.indd 7 11-10-2016 17:40:06