Statistical power, sample size, and their reporting in randomized controlled trials

D. Moher, C. S. Dulberg and G. A. Wells
Clinical Epidemiology Unit, Loeb Medical Research Institute, Ottawa Civic Hospital, Ontario, Canada.

OBJECTIVE--To describe the pattern over time in the level of statistical power and the reporting of sample size calculations in published randomized controlled trials (RCTs) with negative results. DESIGN--Our study was a descriptive survey. Power to detect 25% and 50% relative differences was calculated for the subset of trials with negative results in which a simple two-group parallel design was used. Criteria were developed both to classify trial results as positive or negative and to identify the primary outcomes. Power calculations were based on results from the primary outcomes reported in the trials. POPULATION--We reviewed all 383 RCTs published in JAMA, Lancet, and the New England Journal of Medicine in 1975, 1980, 1985, and 1990. RESULTS--Twenty-seven percent of the 383 RCTs (n = 102) were classified as having negative results. The number of published RCTs more than doubled from 1975 to 1990, with the proportion of trials with negative results remaining fairly stable. Of the simple two-group parallel design trials having negative results with dichotomous or continuous primary outcomes (n = 70), only 16% and 36% had sufficient statistical power (80%) to detect a 25% or 50% relative difference, respectively. These percentages did not consistently increase over time. Overall, only 32% of the trials with negative results reported sample size calculations, but the percentage doing so has improved over time from 0% in 1975 to 43% in 1990. Only 20 of the 102 reports made any statement related to the clinical significance of the observed differences. CONCLUSIONS--Most trials with negative results did not have large enough sample sizes to detect a 25% or a 50% relative difference. This result has not changed over time. Few trials discussed whether the observed differences were clinically important. There are important reasons to change this practice. The reporting of statistical power and sample size also needs to be improved.

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Learning from past mistakes: assessing trial quality, power and eligibility in non-renal systemic lupus erythematosus randomized controlled trials
Yuen and Pope
Rheumatology (Oxford) 2008;47:1367-1372.
ABSTRACT | FULL TEXT  

Metaanalysis of the Efficacy of Sublingual Immunotherapy in the Treatment of Allergic Asthma in Pediatric Patients, 3 to 18 Years of Age
Penagos et al.
Chest 2008;133:599-609.
ABSTRACT | FULL TEXT  

Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo et al.
ptjournal 2008;88:156-175.
ABSTRACT | FULL TEXT  

How can we regulate medicines better?
Garattini and Bertele'
BMJ 2007;335:803-805.
FULL TEXT  

Statistical Power of Negative Randomized Controlled Trials Presented at American Society for Clinical Oncology Annual Meetings
Bedard et al.
JCO 2007;25:3482-3487.
ABSTRACT | FULL TEXT  

Evidence-Based Medicine, Part 2. An Introduction to Critical Appraisal of Articles on Therapy
Cardarelli et al.
JAOA: Journal of the American Osteopathic Association 2007;107:299-303.
ABSTRACT | FULL TEXT  

Communicating With Patients About Chemotherapy Costs
de Lemos
JCO 2007;25:2142-2142.
FULL TEXT  

No Significant Difference ... Says Who?
Trout et al.
Am. J. Neuroradiol. 2007;28:195-197.
FULL TEXT  

Defining the clinical improvement in cancer drug therapy: implications for priority setting in healthcare
de Lemos
J Oncol Pharm Pract 2006;12:91-94.
ABSTRACT  

Reporting of Noninferiority and Equivalence Randomized Trials: An Extension of the CONSORT Statement
Piaggio et al.
JAMA 2006;295:1152-1160.
ABSTRACT | FULL TEXT  

Bias in the Medical Journals: A Commentary
Kopans
Am. J. Roentgenol. 2005;185:176-182.
ABSTRACT | FULL TEXT  

Is Statistical Significance Always Significant?
Koretz
Nutr Clin Pract 2005;20:303-307.
ABSTRACT | FULL TEXT  

The Quality of Randomized Controlled Trials in Major Anesthesiology Journals
Greenfield et al.
Anesth. Analg. 2005;100:1759-1764.
ABSTRACT | FULL TEXT  

Methodological quality and reporting of ethical requirements in phase III cancer trials
Tuech et al.
J. Med. Ethics 2005;31:251-255.
ABSTRACT | FULL TEXT  

Dietary Amelioration of Helicobacter pylori Infection: Design Criteria for a Clinical Trial
Fahey et al.
Cancer Epidemiol. Biomarkers Prev. 2004;13:1610-1616.
ABSTRACT | FULL TEXT  

The Need for Randomized Controlled Trials in Podiatric Medical Research
Porthouse and Torgerson
J. Am. Podiatr. Med. Assoc. 2004;94:221-228.
ABSTRACT | FULL TEXT  

Screening for "Cancer": When is it Valid?--Lessons from the Mammography Experience
Kopans et al.
Radiology 2003;229:319-327.
ABSTRACT | FULL TEXT  

A randomized trial of aspirin on the risk of embolic events in patients with infective endocarditis
Chan et al.
J Am Coll Cardiol 2003;42:775-780.
ABSTRACT | FULL TEXT  

A survey of the statistical power of research in behavioral ecology and animal behavior
Jennions and Moller
Behav Ecol 2003;14:438-445.
ABSTRACT | FULL TEXT  

Sample Size Estimation: How Many Individuals Should Be Studied?
Eng
Radiology 2003;227:309-313.
ABSTRACT | FULL TEXT  

Interpreting Negative Results: Postpartum Interview Position Not Associated With Improved Outcomes
Mabry et al.
Arch Pediatr Adolesc Med 2003;157:333-335.
FULL TEXT  

Clinical trials in orthodontics I: demographic details of clinical trials published in three orthodontic journals between 1989 and 1998
Harrison
J. Orthod. 2003;30:25-30.
ABSTRACT | FULL TEXT  

The Continuing Unethical Conduct of Underpowered Clinical Trials
Halpern et al.
JAMA 2002;288:358-362.
ABSTRACT | FULL TEXT  

Publication Bias in Editorial Decision Making
Olson et al.
JAMA 2002;287:2825-2828.
ABSTRACT | FULL TEXT  

Reporting the clinical importance of randomized controlled trials
Harvey
CMAJ 2002;166:712-712.
FULL TEXT  

How well is the clinical importance of study results reported? An assessment of randomized controlled trials
Chan et al.
CMAJ 2001;165:1197-1202.
ABSTRACT | FULL TEXT  

Sifting the evidence--what's wrong with significance tests?
Sterne et al.
ptjournal 2001;81:1464-1469.
FULL TEXT  

Methodological quality and reporting of ethical requirements in clinical trials
Ruiz-Canela et al.
J. Med. Ethics 2001;27:172-176.
ABSTRACT | FULL TEXT  

Trends in Acute Ischemic Stroke Trials Through the 20th Century
Kidwell et al.
Stroke 2001;32:1349-1359.
ABSTRACT | FULL TEXT  

Have Randomized Controlled Trials of Neuroprotective Drugs Been Underpowered? : An Illustration of Three Statistical Principles
Samsa and Matchar
Stroke 2001;32:669-674.
ABSTRACT | FULL TEXT  

Sifting the evidence{---}what's wrong with significance tests? Another comment on the role of statistical methods
Sterne et al.
BMJ 2001;322:226-231.
FULL TEXT  

Electronic Trial Banks: A Complementary Method for Reporting Randomized Trials
Sim et al.
Med Decis Making 2000;20:440-450.
ABSTRACT  

Bayesian Communication: A Clinically Significant Paradigm for Electronic Publication
Lehmann and Goodman
J. Am. Med. Inform. Assoc. 2000;7:254-266.
ABSTRACT | FULL TEXT  

How Well Are Randomized Controlled Trials Reported in the Dermatology Literature?
Adetugbo and Williams
Arch Dermatol 2000;136:381-385.
ABSTRACT | FULL TEXT  

n-3 Polyunsaturated fatty acid requirements of term infants1
Gibson and Makrides
Am. J. Clin. Nutr. 2000;71:251S-255S.
ABSTRACT | FULL TEXT  

Current Concepts Review - Sample Size and Statistical Power in Clinical Orthopaedic Research
FREEDMAN and BERNSTEIN
JBJS 1999;81:1454-60.
FULL TEXT  

The CONSORT Statement: Will It Lead to Improved Reporting of Clinical Trials in Ophthalmology?
Hawkins
Arch Ophthalmol 1999;117:677-680.
FULL TEXT  

The Breast Cancer Screening Controversy and the National Institutes of Health Consensus Development Conference on Breast Cancer Screening for Women Ages 40–49
Kopans
Radiology 1999;210:4-9.
FULL TEXT  

Vena Caval Filters for the Prevention of Pulmonary Embolism
Chuu et al.
NEJM 1998;339:46-48.
FULL TEXT  

Silent Myocardial Ischemia: Some Good News
Killip
Circulation 1997;95:1992-1993.
FULL TEXT  

Uniform structured formats for scientific communications---how far should we go?
DOHERTY
Ann Rheum Dis 1997;56:81-82.
FULL TEXT  

Univariate and Multivariate Omnibus Hypothesis Tests Selected to Control Type I Error Rates When Population Variances Are Not Necessarily Equal
Coombs et al.
REVIEW OF EDUCATIONAL RESEARCH 1996;66:137-179.
ABSTRACT  

Estimating sample sizes for binary, ordered categorical, and continuous outcomes in two group comparisons
Campbell et al.
BMJ 1995;311:1145-1148.
FULL TEXT  

A Mini-Lesson in Statistics: What Causes Treatment Groups to be Deemed 'Not Statistically Different'?
Martens
J Hum Lact 1995;11:117-121.
ABSTRACT  

Antioxidant Vitamins and Colorectal Adenoma
Burke et al.
NEJM 1994;331:1720-1721.
FULL TEXT