Oral Diseases - 2015 - Atkinson - Design and Interpretation of Clinical Research Studies in Oral Medicine A Brief Review
Oral Diseases - 2015 - Atkinson - Design and Interpretation of Clinical Research Studies in Oral Medicine A Brief Review
Oral Diseases - 2015 - Atkinson - Design and Interpretation of Clinical Research Studies in Oral Medicine A Brief Review
12385
Published 2015. This article is a U.S. Government work and is in the public domain in the USA
All rights reserved
www.wiley.com
ANNIVERSARY REVIEW
The objective of this short review is to help researchers than just the classification of study type, such as cohort
improve the designs of their clinical studies. Also design, clinical trial, and case series. Design in this
included is a discussion of the level of evidence provided paper includes study methodology, such as how disease
by the various clinical research study designs. is measured, how subjects are selected, and how the
Oral Diseases (2016) 22, 87–92 data are analyzed. Implementation will include issues
about how study data are collected, how outcomes are
Keywords: clinical research; epidemiology; oral medicine judged, and how well subjects enrolled in the study are
retained. The good news is that many studies can be
improved without expensive new technologies. We hope
that readers of this review will realize that the most
important ingredient in a successful clinical study is
Introduction adequate planning.
In the last 30 years, the number of publications related to
oral diseases has increased tremendously. The majority Design selection
describe the clinical presentation of a disease such as oral ‘Clinical research’ is defined broadly as patient-oriented
lichen planus, biomarkers related to and risk factors for research that usually involves interaction with research
the disease, and the results of small trials testing therapies subjects to collect data about their health or disease. If
for treatment. To manage the increasing volume of publi- the data are collected from existing records, it is a retro-
cations, systematic reviews are conducted to help clini- spective study. Prospective studies involve collection of
cians and researchers identify the best research to advance data as time moves forward. The research design gener-
patient care. While these reviews first evaluated results ally dictates the level of clinical evidence that is the pro-
from clinical trials, the systematic review process has been duct of the study (Hujoel, 2009). For example, a case
extended to observational studies. An example would be series describing the successful treatment of 20 subjects
the review of the studies investigating the association with disease X with a drug approved for another use
between periodontal disease and cardiovascular disease should not be interpreted as evidence of the drug’s effi-
conducted by the American Heart Association (Lockhart cacy for treatment of disease X. A randomized controlled
et al, 2012). trial is almost always needed to make that conclusion.
When one reads a systematic review, one cannot help Therefore, an appropriate design for a clinical research
but note that many publications are excluded. Often the study should be selected after a research question is con-
reasons for exclusion relate to research design and study ceived. A brief overview of designs and factors to con-
implementation methods (Atkins et al, 2004; The sider with each type is presented below (Manolio, 2002;
Cochrane Collaboration, 2011). This short review will Gordis, 2004).
discuss common weaknesses found in many clinical
studies and provide suggestions for their improvement Case report and case series
(Table 1). We use the word ‘design’ to include more A case report or case series provides a careful description
of a patient or several patients with a common disease or
Correspondence: Jane C. Atkinson, Center for Clinical Research, National syndrome. Examples include descriptions of a patient’s
Institute of Dental and Craniofacial Research, National Institutes of clinical course with an unusual facial infection, the presen-
Health, 6701 Democracy Blvd., Room 634, MSC 4878, Bethesda, MD
20892-4878, USA. Tel: 301 435 7908, Fax: 301 480 5912, E-mail:
tations of cases with oral tuberculosis, a group of patients
[email protected] with fluconazole-resistant candidiasis or an unusual
Received 18 October 2015; accepted 20 October 2015 presentation of allergy in the oral cavity. The description
Design of clinical research studies
JC Atkinson and DB Clark
88
Table 1 Common problems with clinical studies
Population selection Subjects do not represent the full spectrum of persons with the Enroll consecutive patients in a cohort study or use data
disease, disorder or syndrome from consecutive patients in a case series
Referral center bias because academic centers treat patients If the disease of interest is common, one may wish to use
with more severe disease methods to select a random sample
Include community-based centers that treat patients with
better managed disease
Inclusion/exclusion The inclusion/exclusion criteria are overly restrictive or poorly Criteria should be defined clearly and included in the
criteria defined, yielding results that are not generalizable to the entire publication
population of individuals with the disease or cannot be reproduced Criteria should be as broad as possible to not exclude
those most likely to benefit from a new therapy, such
as the elderly
Subject selection Subjects not selected with validated methods. Subjects should be selected using validated diagnostic
methods, realizing that diagnostic criteria are refined
over time
Control selection Controls are not well matched Controls should be matched to controls carefully, such as
by age, SES status and sex/gender
Multiple control groups may be needed, such as a healthy
control and diseased control group
Objective study Methods for judging study outcomes are not validated The methods for measuring outcomes should be validated
outcomes before enrolling subjects. This is critical to generating
clinical science that is reproducible
Study outcomes judged with knowledge of group assignment. This Both clinical outcomes and biological specimen evaluation
includes observational studies and interventional studies should be performed by examiners who have no
knowledge of group assignment or the diagnosis of a
subject
Subjective study Patient-reported outcomes (PRO) are not considered. A statistically When measuring PROs, it is important to use
outcomes significant reduction in the surface area affected by a mucosal questionnaires that are validated for the same reasons it is
disease may not represent a meaningful change to patients important to use validated clinical assessment tools.
A series of validated PRO questionnaires is available
from http://www.nihpromis.org
Analytical methods Post hoc analyses used to analyze data The analytical plan should be established before examining
the data. Post hoc analyses can yield biased,
non-reproducible results
Insufficient numbers of events to test study hypotheses Extend observation to assess the disease or condition
outcome
Poor retention Develop a good retention plan at the outset of the study
Oral Diseases
Design of clinical research studies
JC Atkinson and DB Clark
89
(Table 1). Ideally, complete data need to be collected from received radiation as part of the conditioning regime for
every participant enrolled in the study. Those evaluating stem cell transplant, a limitation of this research study
clinical outcomes such as caries should be calibrated, and would be that children who did not survive the transplant
images such as radiographs or pathology slides should be would not be included in the study population. This may
judged by individuals using predetermined outcomes, with be an important consideration depending on the research
a system to classify abnormalities. Those judging images questions being asked.
or pathological slides should not be aware of a partici- A high-quality case–control study will have the follow-
pant’s diagnosis. The overall goal is to consistently and ing features (Table 1). First, everyone in the study is eval-
independently collect and analyze the research data. Inves- uated using the same methods or instruments. Secondly, if
tigators should be careful to decide what information they possible, those judging clinical outcomes should not know
can collect given their time and resources. It is impossible whether a subject is a case or control. This is important,
to collect everything; instead, focus on collecting the most as investigators may be more likely to judge a certain con-
valuable data using robust methods and collecting com- dition such as mucosal erythema or gingival inflammation
plete data from all research participants. as present if they know the subject is a case. A good
example of a high-quality case–control study was one
Case–control studies started soon after HIV began to present in children. Six
A case–control study is used to determine whether an hundred children born to HIV-positive mothers were eval-
exposure might be related to the development of a certain uated from birth longitudinally by examiners who initially
disease. Cases are individuals with the disease; controls did not know the children’s HIV status (European Collab-
are those without the disease who are carefully matched orative Study). In some cases, both infected and unin-
and drawn from the same population as the cases fected infants had the same mother. This design allowed
(Table 1). The goal is to determine whether an exposure more accurate estimates of the prevalence of HIV compli-
(such as smoking or past infection with mumps virus) is cations in children, such as oral candidiasis and parotid
more prevalent in the cases than the controls to the extent enlargement.
that the exposure can be considered a risk factor or protec- Case–control studies are subject to many types of bias,
tive factor for the development of the disease. including exposure suspicion bias and recall bias. Enrolled
Case–control studies are often used to generate hypothe- participants with the disease (cases) may be seeking a rea-
ses about causality, but should not be interpreted as son for developing the disease, and controls may be less
demonstrating that the risk factor causes the disease. familiar with their medical histories. To minimize the
Case–control studies often collect retrospective data about potential for bias, the case and control groups must be
an exposure (such as past exposure to benzene) and a carefully defined and selected.
one-time assessment of disease severity (such as a com-
plete blood count) and current exposure levels. Other fac- Longitudinal cohort studies
tors to consider when designing a case–control study A longitudinal cohort study is used to collect data on a
include the numbers and appropriateness of controls. Min- certain population prospectively, usually to document dis-
imally, there should be at least the same number of con- ease course or to determine the incidence of a disease and
trols as cases, cases and controls must be evaluated using factors related to its development. They are time and
the same methods, and the cases and controls should come resource intensive, but generate higher quality evidence
from the same populations (Wacholder et al, 1992a,b,c). about exposures and disease development than a case–
Case–control studies are often used to study rarer diseases control or cross-sectional study. For example, the meta-
that would not occur frequently in a large population- bolic syndrome and cancer project, which includes seven
based cross-sectional study. A good example is head and population-based cohorts from Norway, Austria, and Swe-
neck cancer. Case–control designs are used to establish den, enrolled 577 799 adults who were followed on aver-
that smoking and alcohol consumption are major risk fac- age for 12 years. Longitudinal cohort studies may be the
tors for this malignancy (Day et al, 1993; Conway et al, only method to determine whether reduction or continued
2015). Case–control studies typically report associations exposure to a risk factor, such as smoking or alcohol,
between risk factors and the outcome of interest as odds affects the incidence or severity of a disease such as can-
ratios, which are close to relative risk for rare diseases. cer (Stocks et al, 2012). In this situation, it is unethical to
Electronic medical records (EMRs) can be used to randomize individuals in a clinical trial to an arm that
enhance case–control studies. Past patient data that are not would deliberately expose the research participant to
subject to recall bias, such as total head and neck radiation agents known to cause serious adverse outcomes.
dose, may be available from the medical record, and con- The key elements of a good longitudinal study include
trol data from the same population may be extractable. the following:
However, there may be missing or incomplete data in the • Having established intervals for follow-up and evalu-
EMR and the data are typically not collected systemati- ating subjects within a defined interval, such as yearly
cally, so one must be cautious not to overinterpret results. or every 3 months;
Another limitation of a case–control study is that those • Having sufficient follow-up time to detect changes
with the most serious cases may have died from their dis- in the disease of interest or onset of new disease of
ease and are not available for evaluation (Manolio, 2002). interest; and
If an investigator was determining the long-term conse- • Retention of a high percentage of those enrolled
quences of head and neck radiation on adolescents who initially.
Oral Diseases
Design of clinical research studies
JC Atkinson and DB Clark
90
On occasion, a cross-sectional study is continued as a Clinical trials typically are classified into phases (Phase
longitudinal study using a panel design. The same subjects I, II, III, or IV) that indicate their size and stage of devel-
are followed over time, collecting the same measures at opment. A Phase I study is used to establish dose and
specified intervals. An example is the Medical Expenditure safety of a drug or intervention. They are not randomized
Panel Survey (MEPS) (U.S. Department of Health and and treatment allocation is not concealed (i.e., no one is
Human Services, August 21, 2009) that follows a random blinded or masked to the agent the participant is receiv-
subset of NHANES participants for 2 years after comple- ing), and research subjects are given escalating doses of
tion of the NHANES examination. Following these partic- the intervention until the predefined dose limiting toxicity
ipants for 2 years and collecting data at 6-month intervals (DLT) is achieved (Johnson et al, 2007). A Phase I study
allows for a snapshot of medical spending by the popula- could also be used to enroll a small number of subjects
tion with a baseline understanding of their overall health. with a condition (such as ONJ) being treated with a new
Despite the prospective data collection involved, caution surgical technique to determine essential parameters for a
should be taken to avoid drawing causal inference from future randomized trial. This could include determining an
panel surveys. expected healing time, the postoperative complications,
and eligibility criteria for a subsequent study. Once the
Interpretation of longitudinal studies dose and other parameters are established, the next phase
Causality should not be assumed when two conditions (Phase II) begins.
are associated. An example would be cardiovascular dis- Many designs are employed for Phase II trials, which
ease and periodontal disease. While periodontal disease are conducted to determine initial efficacy of a new inter-
may be present more frequently in those with cardiovas- vention. The most straightforward design is a randomized
cular disease that is not sufficient evidence that periodon- controlled trial (RCT) that randomly allocates subjects to a
tal disease causes cardiovascular disease. As the exposure placebo arm or active therapy arm. Ideally, the treatment
precedes the development of disease, longitudinal studies assignment is concealed from the subjects enrolled in the
provide additional evidence that the exposure (such as study, the investigators conducting the study, and those
periodontal disease) causes the disease. However, one conducting the statistical analyses until the final results are
should be careful about interpretation of this evidence, as determined. The goal is to find the superior treatment and
temporal sequence is not the only criterion required to determine whether a future, larger trial is warranted. Phase
establish causation. The case for causality is strengthened II trials may be conducted at a single center or only enroll
in an association study if the incidence of disease a population from one geographic area. They lack general-
increases or decreases relative to the dose of the expo- izability to a larger population of individuals with a dis-
sure (Gordis, 2004). However, changes in disease inci- ease or disorder.
dence that are correlated with exposure could still be The strength of the RCT methodology is the randomiza-
explained by unidentified causal factors. Also, while lon- tion of subjects to compare interventions to each other
gitudinal studies follow individuals prospectively and col- and/or to a non-intervention control group. The random-
lect data from baseline forward using standardized ization procedure presumably controls for hidden factors
methods, some of the data may be retrospective in nat- that could influence outcomes. For example, if a novel
ure. If a longitudinal study followed a group of adoles- treatment for periodontitis was being tested and smokers
cents with and without cleft palate for 5 years to were eligible, randomization should allow for subjects
determine changes in body image and other patient- who smoke to be equally distributed between study arms.
related measures, information about past medical treat- This greatly reduces the chance that smoking status will
ments for both groups would likely be collected from influence any difference that is observed between study
medical records that may be semi-complete (Manolio, arms.
2002). A Phase III study is necessary to establish the efficacy
of an agent and sometimes is referred to as a pivotal trial.
Clinical trials Large Phase III trials usually enroll hundreds to thousands
If a research study is testing an intervention as a treat- of subjects at several clinical centers. While the inclusion/
ment or an improved diagnostic for disease management, exclusion criteria select a uniform subset of individuals
the study is a clinical trial. ‘Intervention’ includes any- with the disease of interest, conducting the trial at several
thing that can alter the course of a disease, such as a centers should enroll a more diverse array of individuals.
pharmaceutical agent, a medical device, a surgical tech- A Phase III trial has different teams of individuals deliver-
nique, a behavioral intervention, or a public health pro- ing the therapy and judging outcomes. While the methods
gram. Clinical trials are a subset of clinical research, and for delivering therapies and judging outcomes are stan-
randomized controlled trials provide the strongest evi- dardized, there is always some variation between centers.
dence for the causal nature of a modifiable factor and a Therefore, if the new agent is determined to have a posi-
disease outcome. Examples would include determining tive effect on the disease across centers, it is more likely
whether an anti-inflammatory agent that decreased auto- that the agent will be beneficial in practice when pre-
antibody levels and other measures of inflammation scribed by many different practitioners. All of these char-
increased the salivary flow rates of individuals with acteristics of Phase III multicenter trials make results more
Sj€ogren’s syndrome, or whether blocking osteoclast activ- generalizable to the population at large.
ity with a bisphosphonate drug prevented osteoporosis A Phase IV trial determines how well an efficacious
progression. treatment works in practice. This could include assessing
Oral Diseases
Design of clinical research studies
JC Atkinson and DB Clark
91
how well the community of patients can comply with a must take into account many factors such as the most
drug schedule, such as taking a drug three times per day, important variables, the distribution of the data, the com-
and the frequency and severity of side effects associated pleteness of the data, and the conclusions that can be
with long-term use of a drug. These studies determine the made given the design of the study. Post hoc data analysis
effectiveness of a therapy. (analyses performed after the original analyses and after
Investigators often fail to consider that establishing evi- investigators know results of the first analyses) must be
dence for a new treatment requires a series of studies. Cir- viewed with caution, especially as the P-values for detect-
cumventing those studies and going straight to an advanced ing significance are often not adjusted for multiple tests
phase clinical trial can increase the probability of a failed and there is a high potential for investigator bias.
trial. For example, the correct dose for a new agent should Another challenging issue in clinical studies is poor
be established with a smaller study that tests different doses retention. If the study intervention is too taxing for partici-
and different lengths of exposure. A drug could have a pants or the primary outcome is collected 2–3 years after
slowly escalating positive effect that might be missed if sub- randomization or a baseline visit, study subjects may drop
jects were not observed long enough in the trial. out of the trial. This threatens the validity and power of
One should also be cautious when reading the results of the study. Investigators must consider their design care-
a Phase II or small clinical trial, even if a statistical differ- fully and realistically to retain as many subjects as possi-
ence is detected. Small trials that are published typically ble for the entire study and have a retention plan in place
overestimate treatment effects (Hennekens and Demets, from study outset.
2009), and small trials that are negative may not be pub- This very brief review highlights important factors to
lished (sometimes referred to as publication bias) (Califf, consider when designing a clinical research study. More
2007). A meta-analysis that combines results from several guidance about the design and implementation of clinical
small clinical trials does not have the generalizability of a studies can be found in the references.
large Phase III trial and may inflate the therapeutic effect
in part because negative trials are not included (LeLorier
Author contributions
et al, 1997).
Questions that one should consider when designing a The authors both contributed to the design, content and
clinical trial include the following: writing of this review.
• Is there enough good evidence from previous studies
to design a clinical trial? Has the natural history, References
including the variability of disease severity across time
been established? Diseases such as oral lichen planus Atkins D, Best D, Briss PA et al (2004). Grading quality
will have periods of quiescence or periods with little of evidence and strength of recommendations. BMJ 328:
pain in the absence of therapy. This variability must 1490.
be known to adequately design an interventional Califf RM (2007). Large clinical trials and registries: clinical
research institutes. In: Gallin JI, Ognibene FP, eds. Principles
study, and can be established with a well-designed and Practice of Clinical Research. 2nd edn. Academic Press:
longitudinal (natural history) cohort study. Burlington, MA, pp. 237–263.
• Was a previous study conducted to establish a dose Cochrane Collaboration (2011). Cochrane Handbook for System-
and duration of therapy? When considering the exist- atic Reviews of Interventions Version 5.1.0.
ing evidence, what should be the appropriate phase or Conway DI, Brenner DR, McMahon AD et al (2015). Estimating
design for the next research question(s)? and explaining the effect of education and income on head
• Can subjects be consistently diagnosed with a disease and neck cancer risk: INHANCE consortium pooled analysis
of 31 case-control studies from 27 countries. Int J Cancer
or condition using accepted, validated criteria?
• Are there existing validated measures to determine the 136: 1125–1139.
Day GL, Blot WJ, Austin DF et al (1993). Racial differences in
outcome variables of interest and can they be applied
risk of oral and pharyngeal cancer: alcohol, tobacco, and other
consistently by all individuals conducting the evalua- determinants. J Natl Cancer Inst 85: 465–473.
tions? Dye BA, Li X, Lewis BG, Iafolla T, Beltran-Aguilar ED, Eke PI
• Does the proposed change of disease activity, such as (2014). Overview and quality assurance for the oral health
a decrease in oral mucosal pain or increase in the component of the National Health and Nutrition Examination
measurement of the oral aperture, translate into a Survey (NHANES), 2009-2010. J Public Health Dent 74:
meaningful change for the patient? Statistical signifi- 248–256.
cance does not necessarily mean clinical significance. European Collaborative Study (1991). Children born to women
• Can the study protocol be implemented consistently with HIV-1 infection: natural history and risk of transmission.
European Collaborative Study. Lancet 337: 253–260.
across all the sites so that each research participant is
treated and evaluated in a similar manner? Gordis L (2004). Epidemiology. Saunders: Philadelphia.
Gottlieb MS, Schroff R, Schanker HM et al (1981). Pneumocys-
tis carinii pneumonia and mucosal candidiasis in previously
Analysis of clinical studies healthy homosexual men: evidence of a new acquired cellular
immunodeficiency. N Engl J Med 305: 1425–1431.
An analysis plan must be developed before the data from Hennekens CH, Demets D (2009). The need for large-scale
a clinical study are analyzed, regardless of the clinical randomized evidence without undue emphasis on small tri-
research design (Table 1). This helps assure that data are als, meta-analyses, or subgroup analyses. JAMA 302: 2361–
analyzed independently and minimizes bias. The plan 2362.
Oral Diseases
Design of clinical research studies
JC Atkinson and DB Clark
92
Hujoel P (2009). Grading the evidence: the core of EBD. J Evid Ruggiero SL, Mehrotra B, Rosenberg TJ, Engroff SL (2004).
Based Dent Pract 9: 122–124. Osteonecrosis of the jaws associated with the use of bisphos-
Johnson LL, Borkowf CB, Albert PS (2007). An introduction to phonates: a review of 63 cases. J Oral Maxillofac Surg 62:
biostatistics: randomization, hypothesis testing and sample size 527–534.
estimation. In: Gallin JI, Ognibene FP, ed. Principles and Stocks T, van Hemelrijck M, Manjer J et al (2012). Blood pressure
Practice of Clinical Research. 2nd ed. Academic Press: and risk of cancer incidence and mortality in the Metabolic Syn-
Burlington, MA, pp. 165–195. drome and Cancer Project. Hypertension 59: 802–810.
LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F U.S. Department of Health and Human Services, A.F.H.R.A.Q.
(1997). Discrepancies between meta-analyses and subsequent August 21, 2009 Medical Expenditure Panel Survey Back-
large randomized, controlled trials. N Engl J Med 337: 536– ground [Online]. Available: http://meps.ahrq.gov/mepsweb/
542. about_meps/survey_back.jsp [accessed on 1 January 2015].
Lockhart PB, Bolger AF, Papapanou PN et al (2012). Periodon- Wacholder S, McLaughlin JK, Silverman DT, Mandel JS
tal disease and atherosclerotic vascular disease: does the evi- (1992a). Selection of controls in case-control studies. I. Princi-
dence support an independent association?: a scientific ples. Am J Epidemiol 135: 1019–1028.
statement from the American Heart Association. Circulation Wacholder S, Silverman DT, McLaughlin JK, Mandel JS
125: 2520–2544. (1992b). Selection of controls in case-control studies. II. Types
Manolio TA (2002). Design and conduct of observational studies of controls. Am J Epidemiol 135: 1029–1041.
and clinical trials. In: Gallin JI, ed. Principles and Practice of Wacholder S, Silverman DT, McLaughlin JK, Mandel JS
Clinical Research. 1st edn. Academic Press: San Diego, pp. (1992c). Selection of controls in case-control studies. III.
187–206. Design options. Am J Epidemiol 135: 1042–1050.
Oral Diseases