Borderline Significance
Borderline Significance
Borderline Significance
Amy Kirkwood and Professor Allan Hackshaw CRUK and UCL Cancer Trials Centre
Examples
Relative risk of 0.75 (95% CI: 0.57-0.99) p-value = 0.048 Clear evidence of an effect? Relative risk of 0.75 (95% CI: 0.55-1.03) p-value = 0.07 No effect?
Size of the treatment effect Eg. hazard ratio, relative risk, absolute risk difference, or mean difference
Size of the standard error, which is influenced by: Number of subjects Number of events* Standard deviation*
Interpretation
Very small p-values (easy to interpret) arise when the effect size is large and the standard error is small. Borderline p-values arise when either: We have a clinically meaningful treatment effect but a moderate or large standard error (usually when there are insufficient participants or events). Or The treatment effect is smaller than expected (should have had a larger trial).
0.77
0.90
Similarly there is a 75% chance that 0.72 and 0.95 contains the true HR.
There is only a 6% chance that the range 1.0 contains the true HR
The conclusion reported in the paper was that the addition of etoposide seemed to be beneficial. This is the only randomised trial of etoposide in these children. The disorder is uncommon: 6.5 years to recruit 492 patients across Europe. Another trial is unlikely. Although the target sample size was exceeded, the treatment effect was smaller than expected (HR 0.83 vs 0.60), which is probably why the result was not statistically significant (i.e. trial was not big enough).
Are these sort of results common and how are they reported?
Journal
BMJ The Lancet JAMA NEJM JNCI JCO
No effect
10 11 3
Example 1
Interventions and patient group Primary endpoint Main result Conclusion reported in the Abstract Those receiving nurse-led intervention had higher scores for quality of life and mood, but did not have improvements in symptom intensity scores
Nurse-led psychoeducational intervention versus usual care for palliative care in patients with advanced cancer
Symptom intensity, assessed by an assessment scale (quality of life and resource use were other endpoints) N=322
Example 2
Interventions and patient group Conclusion reported in the Abstract Admissions to hospital were significantly reducedbut no other clinical benefits were shown
Primary endpoint
Main result
Tailored care plan versus usual care in patients with coronary heart disease
Patients with systolic blood pressure >140mm Hg at 18 months (hospital admission was another endpoint) N=903
Example 3
Interventions and patient group Primary endpoint Main result Conclusion reported in the Abstract
Pre-surgical chemoradiotherapy versus chemotherapy Overall survival in patients with N=126 locally advanced (target was 576) cancer of the esophagogastric junction.
Although statistical significance was not achieved, results point to a survival advantage for preoperative chemoradiotherapy
Example 4
Interventions and patient group Primary endpoint Main result Conclusion reported in the Abstract
Aerobic exercise training plus usual care versus usual care alone, in patients with chronic heart failure
Example 5
Interventions and patient group Primary endpoint Main result Conclusion reported in the Abstract ..a single inexpensive artesunate suppository substantially reduces the risk of death or permanent disability
Artesunate suppository versus placebo in patients with severe malaria who cannot be treated orally; N=12,068
Mortality
Example 6
Interventions and patient group
Telephone counselling using cognitive behavioural skills vs. no intervention to encourage smoking cessation in adolescents; N=2151
Primary endpoint
Main result
Possible solutions
Design trials with larger numbers. But not always feasible (eg high costs or rare disorder) However, even a relatively large trial can produce an effect size smaller than expected (Ewings sarcoma example) Meta analyses. Example (doublet chemotherapy for pancreatic cancer):
One trial: HR 0.86, 95% CI 0.72-1.02, p=0.08 Meta-analysis 3 trials: HR 0.86, 95% CI 0.75-0.98, p=0.02
Conclusions
Borderline results cannot be used as strong evidence either in favour or against an intervention But do not completely dismiss an effect if p>0.05 when the treatment effect is clinically meaningful Do not conclude no effect; look at other endpoints, and other evidence A lack of statistical significance does not mean lack of an effect (Altman & Bland BMJ 1995)
Conclusions
Say that there is probably evidence of an effect but use appropriate language, eg words such as suggestion, indication and seems The same principles apply to other areas of research (eg risk factors)