Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials

K. F. Schulz, I. Chalmers, R. J. Hayes and D. G. Altman
Division of STD/HIV Prevention, Centers for Disease Control and Prevention, Atlanta, GA 30333.

OBJECTIVE--To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects. DESIGN--An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects. DATA SOURCES--Meta-analyses from the Cochrane Pregnancy and Childbirth Database. MAIN OUTCOME MEASURES--The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding. RESULTS--Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P < .001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P = .01), with odds ratios being exaggerated by 17%. CONCLUSIONS--This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials.

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Blinding of Outcomes in Trials of Orthopaedic Trauma: An Opportunity to Enhance the Validity of Clinical Trials
Karanicolas et al.
JBJS 2008;90:1026-1033.
ABSTRACT | FULL TEXT  

Quality of randomized clinical trials in juvenile idiopathic arthritis
Abrahamyan et al.
Rheumatology (Oxford) 2008;47:640-645.
ABSTRACT | FULL TEXT  

Does the Addition of a Second Antipsychotic Drug Improve Clozapine Treatment?
Barbui et al.
Schizophr Bull 2008;0:sbn030v1-sbn030.
ABSTRACT | FULL TEXT  

Internal and external validity of cluster randomised trials: systematic review of recent trials
Eldridge et al.
BMJ 2008;336:876-880.
ABSTRACT | FULL TEXT  

Screening and case-finding instruments for depression: a meta-analysis
Gilbody et al.
CMAJ 2008;178:997-1003.
ABSTRACT | FULL TEXT  

Randomized Clinical Trial of Small-Incision and Laparoscopic Cholecystectomy in Patients With Symptomatic Cholecystolithiasis: Primary and Clinical Outcomes
Keus et al.
Arch Surg 2008;143:371-377.
ABSTRACT | FULL TEXT  

Gynaecologic surgery from uncertainty to science: evolution of randomized control trials
Selman et al.
Hum Reprod 2008;23:827-831.
ABSTRACT | FULL TEXT  

Systematic Review: Comparative Effectiveness and Harms of Treatments for Clinically Localized Prostate Cancer
Wilt et al.
ANN INTERN MED 2008;148:435-448.
ABSTRACT | FULL TEXT  

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
Wood et al.
BMJ 2008;336:601-605.
ABSTRACT | FULL TEXT  

Empirical Evidence of Bias in the Design of Experimental Stroke Studies: A Metaepidemiologic Approach
Crossley et al.
Stroke 2008;39:929-934.
ABSTRACT | FULL TEXT  

Renal Replacement Therapy in Patients With Acute Renal Failure: A Systematic Review
Pannu et al.
JAMA 2008;299:793-805.
ABSTRACT | FULL TEXT  

Extending the CONSORT Statement to Randomized Trials of Nonpharmacologic Treatment: Explanation and Elaboration
Boutron et al.
ANN INTERN MED 2008;148:295-309.
ABSTRACT | FULL TEXT  

RE: LETTER TO THE EDITOR ON "BIAS IN CLINICAL INTERVENTION RESEARCH"
Marmor et al.
Am J Epidemiol 2008;167:500-501.
FULL TEXT  

Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo et al.
ptjournal 2008;88:156-175.
ABSTRACT | FULL TEXT  

Evaluating the Quality of Randomized Controlled Trials that Examine the Efficacy of Natural Health Products: A Systematic Review of Critical Appraisal Instruments
Whelan et al.
Evid Based Complement Alternat Med 2008;0:nem186v1-nem186.
ABSTRACT | FULL TEXT  

A well-conducted randomized trial that establishes no benefit of therapy is an important medical advance
Garg et al.
Nephrol Dial Transplant 2008;23:52-55.
FULL TEXT  

Systematic Review and Meta-analysis: When One Study Is Just not Enough
Garg et al.
CJASN 2008;3:253-260.
FULL TEXT  

The efficacy and harm of prostaglandin analogues for IOP reduction in glaucoma patients compared to dorzolamide and brimonidine: a systematic review
Hodge et al.
Br. J. Ophthalmol. 2008;92:7-12.
ABSTRACT | FULL TEXT  

Considerations of Study Design
Koretz
Nutr Clin Pract 2007;22:593-598.
ABSTRACT | FULL TEXT  

State of the Evidence on Acute Asthma Management in Children: A Critical Appraisal of Systematic Reviews
Boluyt et al.
Pediatrics 2007;120:1334-1343.
ABSTRACT | FULL TEXT  

Diagnosis and Management of Stable Chronic Obstructive Pulmonary Disease: A Clinical Practice Guideline from the American College of Physicians
Qaseem et al.
ANN INTERN MED 2007;147:633-638.
ABSTRACT | FULL TEXT  

Management of Stable Chronic Obstructive Pulmonary Disease: A Systematic Review for a Clinical Practice Guideline
Wilt et al.
ANN INTERN MED 2007;147:639-653.
ABSTRACT | FULL TEXT  

Hormesis and precaution: the twain shall meet
Ricci and MacDonald
Hum Exp Toxicol 2007;26:877-889.
ABSTRACT  

Oral Drug Therapy for Multiple Neglected Tropical Diseases: A Systematic Review
Reddy et al.
JAMA 2007;298:1911-1924.
ABSTRACT | FULL TEXT  

Systematic review of the clinical efficacy and safety of sevelamer in dialysis patients
Tonelli et al.
Nephrol Dial Transplant 2007;22:2856-2866.
ABSTRACT | FULL TEXT  

Quality in epidemiological research: should we be submitting papers before we have the results and submitting more hypothesis-generating research?
Lawlor
Int J Epidemiol 2007;36:940-943.
FULL TEXT  

STROBE: new standards for reporting observational epidemiology, a chance to improve
Ebrahim and Clarke
Int J Epidemiol 2007;36:946-948.
FULL TEXT  

Placebo-controlled trials of Chinese herbal medicine and conventional medicine comparative study
Shang et al.
Int J Epidemiol 2007;36:1086-1092.
ABSTRACT | FULL TEXT  

Erythropoietin-receptor agonists in critically ill patients: a meta-analysis of randomized controlled trials
Zarychanski et al.
CMAJ 2007;177:725-734.
ABSTRACT | FULL TEXT  

Blinding Techniques in Randomized Controlled Trials of Laser Therapy: An Overview and Possible Solution
Relf et al.
Evid Based Complement Alternat Med 2007;0:nem085v1-nem085.
ABSTRACT | FULL TEXT  

Review: Benzodiazepines in generalized anxiety disorder: heterogeneity of outcomes based on a systematic review and meta-analysis of clinical trials
Martin et al.
J Psychopharmacol 2007;21:774-782.
ABSTRACT  

The characteristics of unsolicited clinical oncology literature provided by pharmaceutical industry
Wick et al.
Ann Oncol 2007;18:1580-1582.
FULL TEXT  

Transparency as the Route to Evidence-Informed Professional Education
Gambrill
Research on Social Work Practice 2007;17:553-560.
ABSTRACT  

Is wet combing effective in children with pediculosis capitis infestation?
Arch. Dis. Child. 2007;92:818-820.
FULL TEXT  

Systematic Review: Implantable Cardioverter Defibrillators for Adults with Left Ventricular Systolic Dysfunction
Ezekowitz et al.
ANN INTERN MED 2007;147:251-262.
ABSTRACT | FULL TEXT  

Constraint-induced movement therapy in the treatment of the upper limb in children with hemiplegic cerebral palsy: a Cochrane systematic review
Hoare et al.
Clin Rehabil 2007;21:675-685.
ABSTRACT  

Efficacy and safety of aminoglycoside monotherapy: systematic review and meta-analysis of randomized controlled trials
Vidal et al.
J Antimicrob Chemother 2007;60:247-257.
ABSTRACT | FULL TEXT  

Symposium. How to Participate in Orthopaedic Randomized Clinical Trials
Trippel et al.
JBJS 2007;89:1856-1864.
FULL TEXT  

Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal et al.
Int J Epidemiol 2007;36:847-857.
ABSTRACT | FULL TEXT  

Measurement of long-term outcomes in observational and randomised controlled trials
Hodgson et al.
Br. J. Psychiatry 2007;191:s78-s84.
ABSTRACT | FULL TEXT  

The Record on Rosiglitazone and the Risk of Myocardial Infarction
Psaty and Furberg
NEJM 2007;357:67-69.
FULL TEXT  

Meta-analysis: The Effect of Dietary Counseling for Weight Loss
Dansinger et al.
ANN INTERN MED 2007;147:41-50.
ABSTRACT | FULL TEXT  

Mycophenolate mofetil in induction and maintenance therapy of severe lupus nephritis: a meta-analysis of randomized controlled trials
Zhu et al.
Nephrol Dial Transplant 2007;22:1933-1942.
ABSTRACT | FULL TEXT  

The design and assessment of prospective randomised, controlled trials in orthopaedic surgery
Boutron et al.
J Bone Joint Surg Br 2007;89-B:858-863.
ABSTRACT | FULL TEXT  

Meta-analysis: Acupuncture for Osteoarthritis of the Knee
Manheimer et al.
ANN INTERN MED 2007;146:868-877.
ABSTRACT | FULL TEXT  

A collaborative effort to apply the evidence-based review process to the field of nutrition: challenges, benefits, and lessons learned
Balk et al.
Am. J. Clin. Nutr. 2007;85:1448-1456.
ABSTRACT | FULL TEXT  

Treatment of Sudden Sensorineural Hearing Loss: I. A Systematic Review
Conlin and Parnes
Arch Otolaryngol Head Neck Surg 2007;133:573-581.
ABSTRACT | FULL TEXT  

Systematic Review: Repair of Unruptured Abdominal Aortic Aneurysm
Lederle et al.
ANN INTERN MED 2007;146:735-741.
ABSTRACT | FULL TEXT  

Lessons learned on the path to a healthier brain: dispelling the myths and challenging the hypotheses.
Rubens and Nathan
Perfusion 2007;22:153-160.
ABSTRACT  

Preliminary Examination of a Proposed Treatment-Based Classification System for Patients Receiving Physical Therapy Interventions for Neck Pain
Fritz and Brennan
ptjournal 2007;87:513-524.
ABSTRACT | FULL TEXT  

Oral decontamination for prevention of pneumonia in mechanically ventilated adults: systematic review and meta-analysis
Chan et al.
BMJ 2007;334:889-889.
ABSTRACT | FULL TEXT  

Meta-analysis: Chondroitin for Osteoarthritis of the Knee or Hip
Reichenbach et al.
ANN INTERN MED 2007;146:580-590.
ABSTRACT | FULL TEXT  

Timing of Tourniquet Release in Knee Arthroplasty. Meta-Analysis of Randomized, Controlled Trials
Rama et al.
JBJS 2007;89:699-705.
ABSTRACT | FULL TEXT  

Chronic tendinopathy: effectiveness of eccentric exercise * COMMENTARY 1 * COMMENTARY 2
Woodley et al.
Br. J. Sports. Med. 2007;41:188-198.
FULL TEXT  

The effectiveness and cost effectiveness of a national lay-led self care support programme for patients with long-term conditions: a pragmatic randomised controlled trial
Kennedy et al.
J. Epidemiol. Community Health 2007;61:254-261.
ABSTRACT | FULL TEXT  

Reporting of Outcomes in Orthopaedic Randomized Trials: Does Blinding of Outcome Assessors Matter?
Poolman et al.
JBJS 2007;89:550-558.
ABSTRACT | FULL TEXT  

Mortality in Randomized Trials of Antioxidant Supplements for Primary and Secondary Prevention: Systematic Review and Meta-analysis
Bjelakovic et al.
JAMA 2007;297:842-857.
ABSTRACT | FULL TEXT  

Impact of Adjustment for Quality on Results of Metaanalyses of Diagnostic Accuracy
Leeflang et al.
Clin. Chem. 2007;53:164-172.
ABSTRACT | FULL TEXT  

Efficacy and Safety of Anticoagulant Treatment in Acute Cardioembolic Stroke: A Meta-Analysis of Randomized Controlled Trials
Paciaroni et al.
Stroke 2007;38:423-430.
ABSTRACT | FULL TEXT  

Comparison of treatment effects between animal experiments and clinical trials: systematic review
Perel et al.
BMJ 2007;334:197-197.
ABSTRACT | FULL TEXT  

Acetaminophen and Aminotransferase Elevations
Yuan et al.
JAMA 2006;296:2798-2798.
FULL TEXT  

Respiratory Physiotherapy To Prevent Pulmonary Complications After Abdominal Surgery: A Systematic Review
Pasquina et al.
Chest 2006;130:1887-1899.
ABSTRACT | FULL TEXT  

More Random Thoughts
Reider
Am J Sports Med 2006;34:1889-1890.
FULL TEXT  

Stuttering treatment research 1970-2005: I. Systematic review incorporating trial quality assessment of behavioral, cognitive, and related approaches.
Bothe et al.
AJSLP 2006;15:321-341.
ABSTRACT | FULL TEXT  

Acupuncture for peripheral joint osteoarthritis: A systematic review and meta-analysis
Kwon et al.
Rheumatology (Oxford) 2006;45:1331-1337.
ABSTRACT | FULL TEXT  

Evidence of improving quality of reporting of randomized controlled trials in subfertility
Dias et al.
Hum Reprod 2006;21:2617-2627.
ABSTRACT | FULL TEXT  

Pharmacological prevention of serious anaphylactic reactions due to iodinated contrast media: systematic review
Tramer et al.
BMJ 2006;333:675.
ABSTRACT | FULL TEXT  

Where next for China? Keep to the basic principles of critical appraisal.
Heneghan
BMJ 2006;333:499-499.
FULL TEXT  

Probiotics for pediatric antibiotic-associated diarrhea: a meta-analysis of randomized placebo-controlled trials.
Johnston et al.
CMAJ 2006;175:377-383.
ABSTRACT | FULL TEXT  

Improving the quality of reporting randomized controlled trials in cardiothoracic surgery: The way forward.
Tiruvoipati et al.
J. Thorac. Cardiovasc. Surg. 2006;132:233-240.
ABSTRACT | FULL TEXT  

Nutritional interventions and outcome in patients with cancer or preinvasive lesions: systematic review.
Davies et al.
JNCI J Natl Cancer Inst 2006;98:961-973.
ABSTRACT | FULL TEXT  

Psychological interventions to improve glycaemic control in patients with type 1 diabetes: systematic review and meta-analysis of randomised controlled trials
Winkley et al.
BMJ 2006;333:65.
ABSTRACT | FULL TEXT  

Conscious sedation and analgesia for oocyte retrieval during IVF procedures: a Cochrane review.
Kwan et al.
Hum Reprod 2006;21:1672-1679.
ABSTRACT | FULL TEXT  

When in doubt should we cut it out? The role of surgery in non-small cell lung cancer.
Holty and Gould
Thorax 2006;61:554-556.
FULL TEXT  

Statins for Improving Renal Outcomes: A Meta-Analysis
Sandhu et al.
J. Am. Soc. Nephrol. 2006;17:2006-2016.
ABSTRACT | FULL TEXT  

Vitamin K and the Prevention of Fractures: Systematic Review and Meta-analysis of Randomized Controlled Trials.
Cockayne et al.
Arch Intern Med 2006;166:1256-1261.
ABSTRACT | FULL TEXT  

Meta-Analysis: Effect of Long-Acting {beta}-Agonists on Severe Asthma Exacerbations and Asthma-Related Deaths
Salpeter et al.
ANN INTERN MED 2006;144:904-912.
ABSTRACT | FULL TEXT  

A systematic review and meta-analysis of randomized controlled trials on metformin co-administration during gonadotrophin ovulation induction or IVF in women with polycystic ovary syndrome
Costello et al.
Hum Reprod 2006;21:1387-1399.
ABSTRACT | FULL TEXT  

Commentary: Grading the credibility of molecular evidence for complex diseases
Ioannidis
Int J Epidemiol 2006;35:572-578.
FULL TEXT  

Oral vitamin B12 versus intramuscular vitamin B12 for vitamin B12 deficiency: a systematic review of randomized controlled trials
Butler et al.
Fam Pract 2006;23:279-285.
ABSTRACT | FULL TEXT  

Weighing the evidence to formulate dietary guidelines.
Woolf
J. Am. Coll. Nutr. 2006;25:277S-284S.
ABSTRACT | FULL TEXT  

Body Mass Index and Adenocarcinomas of the Esophagus or Gastric Cardia: A Systematic Review and Meta-analysis.
Kubo and Corley
Cancer Epidemiol. Biomarkers Prev. 2006;15:872-878.
ABSTRACT | FULL TEXT  

Evidence-Based Practice and Policy: Choices Ahead
Gambrill
Research on Social Work Practice 2006;16:338-357.
ABSTRACT