This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on ISI (127)  • Contact me when this article is cited  Related Content  • Related letter  • Related article  • Similar articles in JAMA  Topic Collections  • Evidence-Based Medicine  • Quality of Care, Other  • Randomized Controlled Trial  • Alert me on articles by topic

Ethan M. Balk, MD,MPH; Peter A. L. Bonis, MD; Harry Moskowitz, MD,MS; Christopher H. Schmid, PhD; John P. A. Ioannidis, MD; Chenchen Wang, MD,MSc; Joseph Lau, MD

JAMA. 2002;287:2973-2982.

Context  Specific features of trial quality may be associated with exaggeration or shrinking of the observed treatment effect in randomized studies. Therefore, assessment of trial quality is often used in meta-analysis. However, the degree to which specific quality measures are associated with treatment effects has not been well established across a broad range of clinical areas.

Objective  To determine if quality measures are associated with treatment effect size in randomized controlled trials (RCTs).

Design  Quality measures from published quality assessment scales were evaluated in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery). Included meta-analyses incorporated at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant between-study heterogeneity in the odds ratio (OR) scale.

Main Outcome Measures  Relative ORs comparing overall treatment effect (summary OR) of high vs low-quality studies, as determined by each quality measure, with relative ORs less than 1 indicating larger treatment effect in low-quality studies.

Results  Twenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses. Relative ORs of high vs low-quality studies for these quality measures ranged from 0.83 to 1.26; none was statistically significantly associated with treatment effect. The proportion of studies fulfilling specific quality measures varied widely in the 4 medical areas. In analyses limited to specific medical areas, placebo control, multicenter studies, study country, caregiver blinding, and statistical methods were significantly associated with treatment effect on 7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions of these associations were not consistent.

Conclusions  Individual quality measures are not reliably associated with the strength of treatment effect across studies and medical areas. Although use of specific quality measures may be appropriate in specific well-defined areas in which there is pertinent evidence, findings of associations with treatment effect cannot be generalized to all clinical areas or meta-analyses.

Author Affiliations: Evidence-based Practice Center, Division of Clinical Care Research, Tufts University School of Medicine, New England Medical Center, Boston, Mass (Drs Balk, Bonis, Moskowitz, Schmid, Wang, and Lau); and the Biomedical Research Institute, Foundation for Research and Technology Hellas, Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece (Dr Ioannidis). Dr Moskowitz is now with the Division of General Pediatrics, Mount Sinai Hospital, Mount Sinai School of Medicine, New York, NY.

RELATED LETTER

Allocation Concealment in Clinical Trials
Kenneth F. Schulz, Douglas G. Altman, David Moher, Peter Jüni, Matthias Egger, Ethan M. Balk, Peter A. L. Bonis, Joseph Lau, and John P. A. Ioannidis
JAMA. 2002;288(19):2406-2409.
EXTRACT | FULL TEXT  

RELATED ARTICLE

June 12, 2002
JAMA. 2002;287(22):3017-3018.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Optimal Medical Therapy Is a Proven Option for Chronic Stable Angina
O'Rourke
J Am Coll Cardiol 2008;52:905-907.
ABSTRACT | FULL TEXT  

Interventions to Increase Self-efficacy in the Context of Addiction Behaviours: A Systematic Literature Review
Hyde et al.
J Health Psychol 2008;13:607-623.
ABSTRACT  

Doctors' versus patients' global assessments of treatment effectiveness: empirical survey of diverse treatments in clinical trials
Evangelou et al.
BMJ 2008;336:1287-1290.
ABSTRACT | FULL TEXT  

Evaluation of networks of randomized trials
Salanti et al.
Stat Methods Med Res 2008;17:279-301.
ABSTRACT  

Angiotensin-Converting Enzyme Insertion/Deletion Gene Polymorphic Variant as a Marker of Coronary Artery Disease: A Meta-analysis
Zintzaras et al.
Arch Intern Med 2008;168:1077-1089.
ABSTRACT | FULL TEXT  

Blinding of Outcomes in Trials of Orthopaedic Trauma: An Opportunity to Enhance the Validity of Clinical Trials
Karanicolas et al.
JBJS 2008;90:1026-1033.
ABSTRACT | FULL TEXT  

Cognitive Behavior Therapy for Schizophrenia: Effect Sizes, Clinical Models, and Methodological Rigor
Wykes et al.
Schizophr Bull 2008;34:523-537.
ABSTRACT | FULL TEXT  

A Randomized Trial of Longitudinal Effects of Low-Intensity Responsivity Education/Prelinguistic Milieu Teaching
Warren et al.
JSLHR 2008;51:451-470.
ABSTRACT | FULL TEXT  

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
Wood et al.
BMJ 2008;336:601-605.
ABSTRACT | FULL TEXT  

THE AUTHOR REPLIES
Hemila
Am J Epidemiol 2008;167:501-502.
FULL TEXT  

Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo et al.
ptjournal 2008;88:156-175.
ABSTRACT | FULL TEXT  

Systematic Review and Meta-analysis: When One Study Is Just not Enough
Garg et al.
CJASN 2008;3:253-260.
FULL TEXT  

Should Meta-Analyses of Interventions Include Observational Studies in Addition to Randomized Controlled Trials? A Critical Examination of Underlying Principles
Shrier et al.
Am J Epidemiol 2007;166:1203-1209.
ABSTRACT | FULL TEXT  

Oral Drug Therapy for Multiple Neglected Tropical Diseases: A Systematic Review
Reddy et al.
JAMA 2007;298:1911-1924.
ABSTRACT | FULL TEXT  

Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal et al.
Int J Epidemiol 2007;36:847-857.
ABSTRACT | FULL TEXT  

Meta-analysis: The Effect of Dietary Counseling for Weight Loss
Dansinger et al.
ANN INTERN MED 2007;147:41-50.
ABSTRACT | FULL TEXT  

A collaborative effort to apply the evidence-based review process to the field of nutrition: challenges, benefits, and lessons learned
Balk et al.
Am. J. Clin. Nutr. 2007;85:1448-1456.
ABSTRACT | FULL TEXT  

Reporting of Outcomes in Orthopaedic Randomized Trials: Does Blinding of Outcome Assessors Matter?
Poolman et al.
JBJS 2007;89:550-558.
ABSTRACT | FULL TEXT  

Quality of Reporting of Cancer Prognostic Marker Studies: Association With Reported Prognostic Effect
Kyzas et al.
JNCI J Natl Cancer Inst 2007;99:236-243.
ABSTRACT | FULL TEXT  

Sleep-Disordered Breathing and Blood Pressure in Children: A Meta-analysis
Zintzaras and Kaditis
Arch Pediatr Adolesc Med 2007;161:172-178.
ABSTRACT | FULL TEXT  

Impact of Adjustment for Quality on Results of Metaanalyses of Diagnostic Accuracy
Leeflang et al.
Clin. Chem. 2007;53:164-172.
ABSTRACT | FULL TEXT  

A Systematic Review of School-Based Interventions to Prevent Bullying
Vreeman and Carroll
Arch Pediatr Adolesc Med 2007;161:78-88.
ABSTRACT | FULL TEXT  

Acetaminophen and Aminotransferase Elevations
Yuan et al.
JAMA 2006;296:2798-2798.
FULL TEXT  

'They would say that, wouldn't they?' A reader's guide to author and sponsor biases in clinical research
Bastian
JRSM 2006;99:611-614.
FULL TEXT  

Cochrane reviews v industry supported meta-analyses: has Cochrane really achieved its goals?
Tostad
BMJ 2006;333:916-916.
FULL TEXT  

Preventing pressure ulcers: a systematic review.
Reddy et al.
JAMA 2006;296:974-984.
ABSTRACT | FULL TEXT  

Poverty and Response to Treatment Among Panic Disorder Patients in Primary Care
Roy-Byrne et al.
Am. J. Psychiatry 2006;163:1419-1425.
ABSTRACT | FULL TEXT  

Commentary: Grading the credibility of molecular evidence for complex diseases
Ioannidis
Int J Epidemiol 2006;35:572-578.
FULL TEXT  

Reporting of randomized controlled trials in Hodgkin lymphoma in biomedical journals.
Kober et al.
JNCI J Natl Cancer Inst 2006;98:620-625.
ABSTRACT | FULL TEXT  

Bias in Clinical Intervention Research
Gluud
Am J Epidemiol 2006;163:493-501.
ABSTRACT | FULL TEXT  

Quality of Randomized Controlled Trials Reporting in the Primary Treatment of Brain Tumors
Lai et al.
JCO 2006;24:1136-1144.
ABSTRACT | FULL TEXT  

Insulin Therapy and In-Hospital Mortality in Critically Ill Patients: Systematic Review and Meta-analysis of Randomized Controlled Trials
Pittas et al.
JPEN J Parenter Enteral Nutr 2006;30:164-172.
ABSTRACT | FULL TEXT  

What the surgeon of tomorrow needs to know about evidence-based surgery.
Maier
Arch Surg 2006;141:317-323.
FULL TEXT  

Performance of Antibodies against Tissue Transglutaminase for the Diagnosis of Celiac Disease: Meta-Analysis
Zintzaras and Germenis
CVI 2006;13:187-192.
ABSTRACT | FULL TEXT  

Women Exposed to Intimate Partner Violence: Expectations and Experiences When They Encounter Health Care Professionals: A Meta-analysis of Qualitative Studies
Feder et al.
Arch Intern Med 2006;166:22-37.
ABSTRACT | FULL TEXT  

A Comparison of Regional Versus General Anesthesia for Ambulatory Anesthesia: A Meta-Analysis of Randomized Controlled Trials
Liu et al.
Anesth. Analg. 2005;101:1634-1642.
ABSTRACT | FULL TEXT  

Effects of Adding Ribavirin to Interferon to Treat Chronic Hepatitis C Infection: A Systematic Review and Meta-analysis of Randomized Trials
Brok et al.
Arch Intern Med 2005;165:2206-2212.
ABSTRACT | FULL TEXT  

Methodological Approach: American College of Chest Physicians Guidelines for the Prevention and Management of Postoperative Atrial Fibrillation After Cardiac Surgery
Fleisher et al.
Chest 2005;128:17S-23S.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews of Diagnostic Technologies
Tatsioni et al.
ANN INTERN MED 2005;142:1048-1055.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews That Assess Treatment Harms
Chou and Helfand
ANN INTERN MED 2005;142:1090-1099.
ABSTRACT | FULL TEXT  

Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study
Moja et al.
BMJ 2005;330:1053.
ABSTRACT | FULL TEXT  

Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study
Pildal et al.
BMJ 2005;330:1049.
ABSTRACT | FULL TEXT  

Comparison of Large Versus Smaller Randomized Trials for Mental Health-Related Interventions
Contopoulos-Ioannidis et al.
Am. J. Psychiatry 2005;162:578-584.
ABSTRACT | FULL TEXT  

A Randomized Effectiveness Trial of Cognitive-Behavioral Therapy and Medication for Primary Care Panic Disorder
Roy-Byrne et al.
Arch Gen Psychiatry 2005;62:290-298.
ABSTRACT | FULL TEXT  

Does immunity regulate ejaculate quality and fertility in humans?
Skau and Folstad
Behav Ecol 2005;16:410-416.
ABSTRACT | FULL TEXT  

Commentary: Empirical evidence of attrition bias in clinical trials
Juni and Egger
Int J Epidemiol 2005;34:87-88.
FULL TEXT  

Meta-analysis of parenteral nutrition versus enteral nutrition in patients with acute pancreatitis
Marik and Zaloga
BMJ 2004;328:1407.
ABSTRACT | FULL TEXT  

The Number, Quality, and Coverage of Randomized Controlled Trials in Nephrology
Strippoli et al.
J. Am. Soc. Nephrol. 2004;15:411-419.
ABSTRACT | FULL TEXT  

Efficacy of Postoperative Epidural Analgesia: A Meta-analysis
Block et al.
JAMA 2003;290:2455-2463.
ABSTRACT | FULL TEXT  

Evaluation of Cluster Randomized Controlled Trials in Sub-Saharan Africa
Isaakidis and Ioannidis
Am J Epidemiol 2003;158:921-926.
ABSTRACT | FULL TEXT  

Methodological Differences in Clinical Trials Evaluating Nonpharmacological and Pharmacological Treatments of Hip and Knee Osteoarthritis
Boutron et al.
JAMA 2003;290:1062-1070.
ABSTRACT | FULL TEXT  

Association of Funding and Conclusions in Randomized Drug Trials: A Reflection of Treatment Effect or Adverse Events?
Als-Nielsen et al.
JAMA 2003;290:921-928.
ABSTRACT | FULL TEXT  

Cardiac Resynchronization and Death From Progressive Heart Failure: A Meta-analysis of Randomized Controlled Trials
Bradley et al.
JAMA 2003;289:730-740.
ABSTRACT | FULL TEXT  

Allocation Concealment in Clinical Trials
Schulz et al.
JAMA 2002;288:2406-2409.
FULL TEXT