This Article  • References  • Full text PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Social Bookmarking   What's this?

Dimensions of Methodological Quality Associated With Estimates of Treatment Effects in Controlled Trials

Kenneth F. Schulz, PhD, MBA; Iain Chalmers, MBBS, MSc; Richard J. Hayes, MSc; Douglas G. Altman

JAMA. 1995;273(5):408-412.

Abstract
Objective.
—To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects.

Design.
—An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects.

Data Sources.
—Meta-analyses from the Cochrane Pregnancy and Childbirth Database.

Main Outcome Measures.
—The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding.

Results.
—Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P<.001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P=.01), with odds ratios being exaggerated by 17%.

Conclusions.
—This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials.

(JAMA. 1995;273:408-412)

Author Affiliations
From the Division of STD/HIV Prevention, National Center for Prevention Services, Centers for Disease Control and Prevention, Atlanta, Ga (Dr Schulz); the UK Cochrane Centre, Oxford, England (Drs Schulz and Chalmers); Department of Epidemiology and Population Sciences, London (England) School of Hygiene and Tropical Medicine (Dr Schulz and Mr Hayes); and Medical Statistics Laboratory, Imperial Cancer Research Fund, London (Mr Altman).

Footnotes
Presented in part at the 15th annual meeting of the Society for Clinical Trials, Houston, Tex, May 10, 1994, and at the Second International Cochrane Colloquium, Hamilton, Ontario, October 3, 1994.

Reprint requests to Division of STD/HIV Prevention, NCPS, Mailstop E-02, Centers for Disease Control and Prevention, Atlanta, GA 30333 (Dr Schulz).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Interpretive Bias in Acupuncture Research?: A Case Study
O'Connell et al.
Eval Health Prof 2009;32:393-409.
ABSTRACT  

Evaluating the Quality of Randomized Controlled Trials that Examine the Efficacy of Natural Health Products: A Systematic Review of Critical Appraisal Instruments
Whelan et al.
Evid Based Complement Alternat Med 2009;6:441-448.
ABSTRACT | FULL TEXT  

Are Positive Alternative Medical Therapy Trials Credible?: Evidence From Four High-Impact Medical Journals
Barker Bausell
Eval Health Prof 2009;32:349-369.
ABSTRACT  

Dietary protein and bone health: a systematic review and meta-analysis
Darling et al.
Am. J. Clin. Nutr. 2009;90:1674-1692.
ABSTRACT | FULL TEXT  

Efficacy of Sexually Transmitted Disease/Human Immunodeficiency Virus Sexual Risk-Reduction Intervention for African American Adolescent Females Seeking Sexual Health Services: A Randomized Controlled Trial
DiClemente et al.
Arch Pediatr Adolesc Med 2009;163:1112-1121.
ABSTRACT | FULL TEXT  

Author Response
Costa et al.
ptjournal 2009;89:1289-1291.
FULL TEXT  

Getting a better grip on research: A simple system that works
Maskrey et al.
InnovAiT 2009;2:739-749.
ABSTRACT | FULL TEXT  

Motor Control Exercise for Chronic Low Back Pain: A Randomized Placebo-Controlled Trial
Costa et al.
ptjournal 2009;89:1275-1286.
ABSTRACT | FULL TEXT  

Ethical Considerations in Cross-Linguistic Nursing
Carnevale et al.
Nurs Ethics 2009;16:813-826.
ABSTRACT  

Quality of randomised trials in COPD
Bausch et al.
Eur Respir J 2009;34:1060-1065.
ABSTRACT | FULL TEXT  

Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling et al.
BMJ 2009;339:b4012-b4012.
ABSTRACT | FULL TEXT  

Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses
Ioannidis
CMAJ 2009;181:488-493.
FULL TEXT  

NIDCAP: New Controversial Evidence for Its Effectiveness
Ohlsson
Pediatrics 2009;124:1213-1215.
FULL TEXT  

Fall prevention with supplemental and active forms of vitamin D: a meta-analysis of randomised controlled trials
Bischoff-Ferrari et al.
BMJ 2009;339:b3692-b3692.
ABSTRACT | FULL TEXT  

The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study
Nuesch et al.
BMJ 2009;339:b3244-b3244.
ABSTRACT | FULL TEXT  

Randomized Clinical Trials of Constitutional Acupuncture: A Systematic Review
Lee et al.
Evid Based Complement Alternat Med 2009;6:59-64.
ABSTRACT | FULL TEXT  

The devil is in the details...or not? A primer on individual patient data meta-analysis
Sud and Douketis
Evid. Based Med. 2009;14:100-101.
FULL TEXT  

Compliance with and short-term adverse events from clarithromycin versus placebo in patients with stable coronary heart disease: the CLARICOR trial
Jespersen et al.
J Antimicrob Chemother 2009;64:411-415.
ABSTRACT | FULL TEXT  

Residential alternatives to acute psychiatric hospital admission: systematic review
Lloyd-Evans et al.
Br. J. Psychiatry 2009;195:109-117.
ABSTRACT | FULL TEXT  

Is tai chi beneficial for improving aerobic capacity? A systematic review
Lee et al.
Br. J. Sports. Med. 2009;43:569-573.
ABSTRACT | FULL TEXT  

Value of internal limiting membrane peeling in surgery for idiopathic macular hole stage 2 and 3: a randomised clinical trial
Christensen et al.
Br J Ophthalmol 2009;93:1005-1015.
ABSTRACT | FULL TEXT  

North American (Panax quinquefolius) and Asian Ginseng (Panax ginseng) Preparations for Prevention of the Common Cold in Healthy Adults: A Systematic Review
Seida et al.
Evid Based Complement Alternat Med 2009;0:nep068v1-nep068.
ABSTRACT | FULL TEXT  

A Guide to Interpretation of Studies Investigating Subgroups of Responders to Physical Therapy Interventions
Hancock et al.
ptjournal 2009;89:698-704.
ABSTRACT | FULL TEXT  

Enteral Nutrition: A Hard Look at Some Soft Evidence
Koretz
Nutr Clin Pract 2009;24:316-324.
ABSTRACT | FULL TEXT  

Comparing Ultrasound Biometry with Partial Coherence Interferometry for Intraocular Lens Power Calculations: A Randomized Study
Raymond et al.
IOVS 2009;50:2547-2552.
ABSTRACT | FULL TEXT  

Exercise and Health-Related Quality of Life in Older Community-Dwelling Adults: A Meta-Analysis of Randomized Controlled Trials
Kelley et al.
Journal of Applied Gerontology 2009;28:369-394.
ABSTRACT  

Benefits and harms of erythropoiesis-stimulating agents for anemia related to cancer: a meta-analysis
Tonelli et al.
CMAJ 2009;180:E62-E71.
ABSTRACT | FULL TEXT  

Decontamination of the Digestive Tract in ICU Patients
Meessen et al.
NEJM 2009;360:2138-2141.
FULL TEXT  

Clinical Guidelines for the Use of Parenteral and Enteral Nutrition in Adult and Pediatric Patients, 2009
American Society for Parenteral and Enteral Nutrit
JPEN J Parenter Enteral Nutr 2009;33:255-259.
FULL TEXT  

Quality of Reporting of Randomized Clinical Trials in Tai Chi Interventions--A Systematic Review
Li et al.
Evid Based Complement Alternat Med 2009;0:nep022v1-nep022.
ABSTRACT | FULL TEXT  

Review: Methodological quality in medical evidence, quo vadis?
Diaz-Insua
Therapeutic Advances in Urology 2009;1:51-59.
ABSTRACT  

Reporting of systematic reviews of micronutrients and health: a critical appraisal
Chung et al.
Am. J. Clin. Nutr. 2009;89:1099-1113.
ABSTRACT | FULL TEXT  

Does the Addition of a Second Antipsychotic Drug Improve Clozapine Treatment?
Barbui et al.
Schizophr Bull 2009;35:458-468.
ABSTRACT | FULL TEXT  

Ultrasound guidance compared with electrical neurostimulation for peripheral nerve block: a systematic review and meta-analysis of randomized controlled trials
Abrahams et al.
Br J Anaesth 2009;102:408-417.
ABSTRACT | FULL TEXT  

Rituximab Maintenance for the Treatment of Patients With Follicular Lymphoma: Systematic Review and Meta-analysis of Randomized Trials
Vidal et al.
JNCI J Natl Cancer Inst 2009;101:248-255.
ABSTRACT | FULL TEXT  

Efficacy and safety of insulin analogues for the management of diabetes mellitus: a meta-analysis
Singh et al.
CMAJ 2009;180:385-397.
ABSTRACT | FULL TEXT  

Relation of study quality, concordance, take home message, funding, and impact in studies of influenza vaccines: systematic review
Jefferson et al.
BMJ 2009;338:b354-b354.
ABSTRACT | FULL TEXT  

Aldosterone blockade and left ventricular dysfunction: a systematic review of randomized clinical trials
Ezekowitz and McAlister
Eur Heart J 2009;30:469-477.
ABSTRACT | FULL TEXT  

Probiotics, Critical Illness, and Methodologic Bias
Koretz
Nutr Clin Pract 2009;24:45-49.
ABSTRACT | FULL TEXT  

Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?
Thorlund et al.
Int J Epidemiol 2009;38:276-286.
ABSTRACT | FULL TEXT  

Apparently conclusive meta-analyses may be inconclusive--Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses
Brok et al.
Int J Epidemiol 2009;38:287-298.
ABSTRACT | FULL TEXT  

Prevention and treatment of renal disease in Henoch-Schonlein purpura: a systematic review
Chartapisak et al.
Arch. Dis. Child. 2009;94:132-137.
ABSTRACT | FULL TEXT  

Medical Statistics/Test-Taking Strategies
Kamp
ACCP Pulmonary Med Brd Rev 2009;25:295-308.
FULL TEXT  

Study Quality Assessment in Systematic Reviews of Research on Intervention Effects
Wells and Littell
Research on Social Work Practice 2009;19:52-62.
ABSTRACT  

Systematic Reviews of Pharmacy Practice Research: Methodologic Issues in Searching, Evaluating, Interpreting, and Disseminating Results
Charrois et al.
The Annals of Pharmacotherapy 2009;43:118-122.
ABSTRACT | FULL TEXT  

Efficacy of food fortification on serum 25-hydroxyvitamin D concentrations: systematic review
O'Donnell et al.
Am. J. Clin. Nutr. 2008;88:1528-1534.
ABSTRACT | FULL TEXT  

Blinding Techniques in Randomized Controlled Trials of Laser Therapy: An Overview and Possible Solution
Relf et al.
Evid Based Complement Alternat Med 2008;5:383-389.
ABSTRACT | FULL TEXT  

Treatment Interventions in Nursing Home Residents With Urinary Incontinence: A Systematic Review of Randomized Trials
Fink et al.
Mayo Clin Proc. 2008;83:1332-1343.
ABSTRACT | FULL TEXT  

The Effect of Thickened-Feed Interventions on Gastroesophageal Reflux in Infants: Systematic Review and Meta-analysis of Randomized, Controlled Trials
Horvath et al.
Pediatrics 2008;122:e1268-e1277.
ABSTRACT | FULL TEXT  

Amitriptyline in the treatment of fibromyalgia: a systematic review of its efficacy
Nishishinya et al.
Rheumatology (Oxford) 2008;47:1741-1746.
ABSTRACT | FULL TEXT  

Primary Prevention of Cardiovascular Mortality and Events With Statin Treatments A Network Meta-Analysis Involving More Than 65,000 Patients.
Mills et al.
J Am Coll Cardiol 2008;52:1769-1781.
ABSTRACT | FULL TEXT  

Issues related to the conduct of systematic reviews: a focus on the nutrition field
Moher and Tricco
Am. J. Clin. Nutr. 2008;88:1191-1199.
ABSTRACT | FULL TEXT  

The Use of an Inflammation-Modulating Diet in Patients With Acute Lung Injury or Acute Respiratory Distress Syndrome: A Meta-Analysis of Outcome Data
Pontes-Arruda et al.
JPEN J Parenter Enteral Nutr 2008;32:596-605.
ABSTRACT | FULL TEXT  

Clinical outcomes in randomized trials of off- vs. on-pump coronary artery bypass surgery: systematic review with meta-analyses and trial sequential analyses
Moller et al.
Eur Heart J 2008;29:2601-2616.
ABSTRACT | FULL TEXT  

Quality of Reporting of Randomized Controlled Trials in General Endocrinology Literature
Rios et al.
J. Clin. Endocrinol. Metab. 2008;93:3810-3816.
ABSTRACT | FULL TEXT  

Systematic Review: The Trousseau Syndrome Revisited: Should We Screen Extensively for Cancer in Patients with Venous Thromboembolism?
Carrier et al.
ANN INTERN MED 2008;149:323-333.
ABSTRACT | FULL TEXT  

Use of helium-oxygen mixtures in the treatment of croup: a systematic review
Vorwerk and Coats
Emerg. Med. J. 2008;25:547-550.
ABSTRACT | FULL TEXT  

A Systematic Review of Randomized Trials Evaluating Regional Techniques for Postthoracotomy Analgesia
Joshi et al.
Anesth. Analg. 2008;107:1026-1040.
ABSTRACT | FULL TEXT  

Summary of evidence-based review on vitamin D efficacy and safety in relation to bone health
Cranney et al.
Am. J. Clin. Nutr. 2008;88:513S-519S.
ABSTRACT | FULL TEXT  

Comment on: * Early Enteral Supplementation With Key Pharmaconutrients Improves Sequential Organ Failure Assessment Score in Critically Ill Patients With Sepsis: Outcome of a Randomized, Controlled, Double-Blind Trial
Koretz
Nutr Clin Pract 2008;23:447-449.
ABSTRACT | FULL TEXT  

Does exercise training during pregnancy affect gestational age? A randomised controlled trial
Barakat et al.
Br. J. Sports. Med. 2008;42:674-678.
ABSTRACT | FULL TEXT  

Interventions to Increase Self-efficacy in the Context of Addiction Behaviours: A Systematic Literature Review
Hyde et al.
J Health Psychol 2008;13:607-623.
ABSTRACT  

The Enteral vs Parenteral Nutrition Debate Revisited
Thomson
JPEN J Parenter Enteral Nutr 2008;32:474-481.
ABSTRACT | FULL TEXT  

Carts, Horses, and Evidence
Koretz
JPEN J Parenter Enteral Nutr 2008;32:482-485.
FULL TEXT  

Alternative approaches to endoscopic ablation for benign enlargement of the prostate: systematic review of randomised controlled trials
Lourenco et al.
BMJ 2008;337:a449-a449.
ABSTRACT | FULL TEXT  

Cognitive-behavioural therapy for severe mental disorders: back to the future?
Scott
Br. J. Psychiatry 2008;192:401-403.
ABSTRACT | FULL TEXT  

Blinding of Outcomes in Trials of Orthopaedic Trauma: An Opportunity to Enhance the Validity of Clinical Trials
Karanicolas et al.
JBJS 2008;90:1026-1033.
ABSTRACT | FULL TEXT  

Quality of randomized clinical trials in juvenile idiopathic arthritis
Abrahamyan et al.
Rheumatology (Oxford) 2008;47:640-645.
ABSTRACT | FULL TEXT  

Internal and external validity of cluster randomised trials: systematic review of recent trials
Eldridge et al.
BMJ 2008;336:876-880.
ABSTRACT | FULL TEXT  

Screening and case-finding instruments for depression: a meta-analysis
Gilbody et al.
CMAJ 2008;178:997-1003.
ABSTRACT | FULL TEXT  

Randomized Clinical Trial of Small-Incision and Laparoscopic Cholecystectomy in Patients With Symptomatic Cholecystolithiasis: Primary and Clinical Outcomes
Keus et al.
Arch Surg 2008;143:371-377.
ABSTRACT | FULL TEXT  

Gynaecologic surgery from uncertainty to science: evolution of randomized control trials
Selman et al.
Hum Reprod 2008;23:827-831.
ABSTRACT | FULL TEXT  

Systematic Review: Comparative Effectiveness and Harms of Treatments for Clinically Localized Prostate Cancer
Wilt et al.
ANN INTERN MED 2008;148:435-448.
ABSTRACT | FULL TEXT  

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
Wood et al.
BMJ 2008;336:601-605.
ABSTRACT | FULL TEXT  

Empirical Evidence of Bias in the Design of Experimental Stroke Studies: A Metaepidemiologic Approach
Crossley et al.
Stroke 2008;39:929-934.
ABSTRACT | FULL TEXT  

Renal Replacement Therapy in Patients With Acute Renal Failure: A Systematic Review
Pannu et al.
JAMA 2008;299:793-805.
ABSTRACT | FULL TEXT  

Extending the CONSORT Statement to Randomized Trials of Nonpharmacologic Treatment: Explanation and Elaboration
Boutron et al.
ANN INTERN MED 2008;148:295-309.
ABSTRACT | FULL TEXT  

RE: LETTER TO THE EDITOR ON "BIAS IN CLINICAL INTERVENTION RESEARCH"
Marmor et al.
Am J Epidemiol 2008;167:500-501.
FULL TEXT  

Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo et al.
ptjournal 2008;88:156-175.
ABSTRACT | FULL TEXT  

A well-conducted randomized trial that establishes no benefit of therapy is an important medical advance
Garg et al.
Nephrol Dial Transplant 2008;23:52-55.
FULL TEXT  

Systematic Review and Meta-analysis: When One Study Is Just not Enough
Garg et al.
CJASN 2008;3:253-260.
FULL TEXT  

The efficacy and harm of prostaglandin analogues for IOP reduction in glaucoma patients compared to dorzolamide and brimonidine: a systematic review
Hodge et al.
Br J Ophthalmol 2008;92:7-12.
ABSTRACT | FULL TEXT