This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on ISI (437)  • Contact me when this article is cited  Related Content  • Related letters  • Related articles  • Similar articles in JAMA  Topic Collections  • Evidence-Based Medicine  • Quality of Care, Other  • Statistics and Research Methods  • Randomized Controlled Trial  • Alert me on articles by topic

Peter Jüni, MD; Anne Witschi, MD; Ralph Bloch, MD, PhD; Matthias Egger, MD, MSc

JAMA. 1999;282:1054-1060.

Context  Although it is widely recommended that clinical trials undergo some type of quality review, the number and variety of quality assessment scales that exist make it unclear how to achieve the best assessment.

Objective  To determine whether the type of quality assessment scale used affects the conclusions of meta-analytic studies.

Design and Setting  Meta-analysis of 17 trials comparing low-molecular-weight heparin (LMWH) with standard heparin for prevention of postoperative thrombosis using 25 different scales to identify high-quality trials. The association between treatment effect and summary scores and the association with 3 key domains (concealment of treatment allocation, blinding of outcome assessment, and handling of withdrawals) were examined in regression models.

Main Outcome Measure  Pooled relative risks of deep vein thrombosis with LMWH vs standard heparin in high-quality vs low-quality trials as determined by 25 quality scales.

Results  Pooled relative risks from high-quality trials ranged from 0.63 (95% confidence interval [CI], 0.44-0.90) to 0.90 (95% CI, 0.67-1.21) vs 0.52 (95% CI, 0.24-1.09) to 1.13 (95% CI, 0.70-1.82) for low-quality trials. For 6 scales, relative risks of high-quality trials were close to unity, indicating that LMWH was not significantly superior to standard heparin, whereas low-quality trials showed better protection with LMWH (P<.05). Seven scales showed the opposite: high quality trials showed an effect whereas low quality trials did not. For the remaining 12 scales, effect estimates were similar in the 2 quality strata. In regression analysis, summary quality scores were not significantly associated with treatment effects. There was no significant association of treatment effects with allocation concealment and handling of withdrawals. Open outcome assessment, however, influenced effect size with the effect of LMWH, on average, being exaggerated by 35% (95% CI, 1%-57%; P=.046).

Conclusions  Our data indicate that the use of summary scores to identify trials of high quality is problematic. Relevant methodological aspects should be assessed individually and their influence on effect sizes explored.

Author Affiliations: Clinical Epidemiology Study Group, Berne, Switzerland (Drs Jüni and Witschi), Institute for Medical Education, University of Berne (Drs Bloch and Jüni); and MRC Health Services Research Collaboration, Department of Social Medicine, University of Bristol, Bristol, England (Dr Egger).

RELATED LETTERS

Scoring the Quality of Clinical Trials
Willem J. J. Assendelft, Bart W. Koes, Maurits W. van Tulder, Lex M. Bouter, Gerben ter Riet, Pieter Leffers, Maurice Zeegers, Jan P. Vandenbroucke, Peter Jüni, and Matthias Egger
JAMA. 2000;283(11):1421-1423.
EXTRACT | FULL TEXT  

The Ethics of Underpowered Clinical Trials
Janine E. Janosky, John R. Hughes, Richard J. Lilford, Scott D. Halpern, Jason H. T. Karlawish, and Jesse A. Berlin
JAMA. 2002;288(17):2118-2119.
EXTRACT | FULL TEXT  

Allocation Concealment in Clinical Trials
Kenneth F. Schulz, Douglas G. Altman, David Moher, Peter Jüni, Matthias Egger, Ethan M. Balk, Peter A. L. Bonis, Joseph Lau, and John P. A. Ioannidis
JAMA. 2002;288(19):2406-2409.
EXTRACT | FULL TEXT  

Prematurity and Later Cognitive Outcomes
Lex W. Doyle, Saroj Saigal, David L. Streiner, Adnan T. Bhutta, Mario Cleves, Patrick H. Casey, and K. J. S. Anand
JAMA. 2002;288(20):2542-2543.
EXTRACT | FULL TEXT  

RELATED ARTICLES

Empirical Evidence of Design-Related Bias in Studies of Diagnostic Tests
Jeroen G. Lijmer, Ben Willem Mol, Siem Heisterkamp, Gouke J. Bonsel, Martin H. Prins, Jan H. P. van der Meulen, and Patrick M. M. Bossuyt
JAMA. 1999;282(11):1061-1066.
ABSTRACT | FULL TEXT  

Measuring the Quality of Trials: The Quality of Quality Scales
Jesse A. Berlin and Drummond Rennie
JAMA. 1999;282(11):1083-1085.
EXTRACT | FULL TEXT  

September 15, 1999
JAMA. 1999;282(11):1099-1100.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Amitriptyline in the treatment of fibromyalgia: a systematic review of its efficacy
Nishishinya et al.
Rheumatology (Oxford) 2008;0:ken317v1-ken317.
ABSTRACT | FULL TEXT  

Overviews and Systematic Reviews on Low Back Pain
Chou et al.
ANN INTERN MED 2008;148:791-792.
FULL TEXT  

Cognitive Behavior Therapy for Schizophrenia: Effect Sizes, Clinical Models, and Methodological Rigor
Wykes et al.
Schizophr Bull 2008;34:523-537.
ABSTRACT | FULL TEXT  

Use of Metformin in Polycystic Ovary Syndrome: A Meta-Analysis
Creanga et al.
Obstet Gynecol 2008;111:959-968.
ABSTRACT | FULL TEXT  

Empirical Evidence of Bias in the Design of Experimental Stroke Studies: A Metaepidemiologic Approach
Crossley et al.
Stroke 2008;39:929-934.
ABSTRACT | FULL TEXT  

Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo et al.
ptjournal 2008;88:156-175.
ABSTRACT | FULL TEXT  

Perspectives on Evidence-Based Research in Education--What Works? Issues in Synthesizing Educational Program Evaluations
Slavin
EDUCATIONAL RESEARCHER 2008;37:5-14.
ABSTRACT | FULL TEXT  

Oral Drug Therapy for Multiple Neglected Tropical Diseases: A Systematic Review
Reddy et al.
JAMA 2007;298:1911-1924.
ABSTRACT | FULL TEXT  

A Meta-Analysis of 16 Randomized Trials of Sirolimus-Eluting Stents Versus Paclitaxel-Eluting Stents in Patients With Coronary Artery Disease
Schomig et al.
J Am Coll Cardiol 2007;50:1373-1380.
ABSTRACT | FULL TEXT  

The Regression Effect as a Neglected Source of Bias in Nonrandomized Intervention Trials and Systematic Reviews of Observational Studies
Weeks
Eval Health Prof 2007;30:254-265.
ABSTRACT  

Meta-analysis of APOE genotype and subarachnoid hemorrhage: Clinical outcome and delayed ischemia
Lanterna et al.
Neurology 2007;69:766-775.
ABSTRACT | FULL TEXT  

Statins and the Risk of Colorectal Cancer: A Meta-Analysis of 18 Studies Involving More Than 1.5 Million Patients
Bonovas et al.
JCO 2007;25:3462-3468.
ABSTRACT | FULL TEXT  

Are you reading what we are reading? The effect of who interprets medical images on estimates of diagnostic test accuracy in systematic reviews
Brealey and Westwood
Br. J. Radiol. 2007;80:674-677.
ABSTRACT | FULL TEXT  

Thyroid Dysfunction and Effects on Coagulation and Fibrinolysis: A Systematic Review
Squizzato et al.
J. Clin. Endocrinol. Metab. 2007;92:2415-2420.
ABSTRACT | FULL TEXT  

A Systematic Review of the Performance Characteristics of Clinical Event Monitor Signals Used to Detect Adverse Drug Events in the Hospital Setting
Handler et al.
J. Am. Med. Inform. Assoc. 2007;14:451-458.
ABSTRACT | FULL TEXT  

Commentary: Assessing the quality of observational studies--or a lesson from Mars
von Elm
Int J Epidemiol 2007;0:dym112v1-2.
FULL TEXT  

Meta-analysis: Acupuncture for Osteoarthritis of the Knee
Manheimer et al.
ANN INTERN MED 2007;146:868-877.
ABSTRACT | FULL TEXT  

A collaborative effort to apply the evidence-based review process to the field of nutrition: challenges, benefits, and lessons learned
Balk et al.
Am. J. Clin. Nutr. 2007;85:1448-1456.
ABSTRACT | FULL TEXT  

Sensory re-education after nerve injury of the upper limb: a systematic review
Oud et al.
Clin Rehabil 2007;21:483-494.
ABSTRACT  

The Pylorus: Take It or Leave It? Systematic Review and Meta-Analysis of Pylorus-Preserving versus Standard Whipple Pancreaticoduodenectomy for Pancreatic or Periampullary Cancer
Karanicolas et al.
Ann. Surg. Oncol. 2007;14:1825-1834.
ABSTRACT | FULL TEXT  

Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding
Hrobjartsson et al.
Int J Epidemiol 2007;36:654-663.
ABSTRACT | FULL TEXT  

Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
Sanderson et al.
Int J Epidemiol 2007;36:666-676.
ABSTRACT | FULL TEXT  

Meta-analysis: Chondroitin for Osteoarthritis of the Knee or Hip
Reichenbach et al.
ANN INTERN MED 2007;146:580-590.
ABSTRACT | FULL TEXT  

Health technology assessment in obstetrics and gynaecology. Part 1: an overview of the process
Abedin et al.
The Obstetrician and Gynaecologist 2007;9:109-115.
ABSTRACT | FULL TEXT  

Parenting programmes for preventing tobacco, alcohol or drugs misuse in children <18: a systematic review
Petrie et al.
Health Educ Res 2007;22:177-191.
ABSTRACT | FULL TEXT  

Eligibility Criteria of Randomized Controlled Trials Published in High-Impact General Medical Journals: A Systematic Sampling Review
Van Spall et al.
JAMA 2007;297:1233-1240.
ABSTRACT | FULL TEXT  

Reporting of Outcomes in Orthopaedic Randomized Trials: Does Blinding of Outcome Assessors Matter?
Poolman et al.
JBJS 2007;89:550-558.
ABSTRACT | FULL TEXT  

Does pravastatin promote cancer in elderly patients? A meta-analysis
Bonovas and Sitaras
CMAJ 2007;176:649-654.
ABSTRACT | FULL TEXT  

Meta-analysis: Anticoagulant Prophylaxis to Prevent Symptomatic Venous Thromboembolism in Hospitalized Medical Patients
Dentali et al.
ANN INTERN MED 2007;146:278-288.
ABSTRACT | FULL TEXT  

Quality of Reporting of Cancer Prognostic Marker Studies: Association With Reported Prognostic Effect
Kyzas et al.
JNCI J Natl Cancer Inst 2007;99:236-243.
ABSTRACT | FULL TEXT  

Sleep-Disordered Breathing and Blood Pressure in Children: A Meta-analysis
Zintzaras and Kaditis
Arch Pediatr Adolesc Med 2007;161:172-178.
ABSTRACT | FULL TEXT  

Impact of Adjustment for Quality on Results of Metaanalyses of Diagnostic Accuracy
Leeflang et al.
Clin. Chem. 2007;53:164-172.
ABSTRACT | FULL TEXT  

Incorporating Quality of Evidence into Decision Analytic Modeling
Braithwaite et al.
ANN INTERN MED 2007;146:133-141.
ABSTRACT | FULL TEXT  

Economic evaluations of single- versus double-embryo transfer in IVF
Fiddelers et al.
Hum Reprod Update 2007;13:5-13.
ABSTRACT | FULL TEXT  

Mortality in randomized controlled trials comparing drug-eluting vs. bare metal stents in coronary artery disease: a meta-analysis
Nordmann et al.
Eur Heart J 2006;27:2784-2814.
ABSTRACT | FULL TEXT  

A systematic review of cephalometric facial soft tissue changes with the Activator and Bionator appliances in Class II division 1 subjects
Flores-Mir and Major
Eur J Orthod 2006;28:586-593.
ABSTRACT | FULL TEXT  

Intravenous thrombolysis in stroke patients of >=80 versus <80 years of age--a systematic review across cohort studies.
Engelter et al.
Age Ageing 2006;35:572-580.
ABSTRACT | FULL TEXT  

Acupuncture for peripheral joint osteoarthritis: A systematic review and meta-analysis
Kwon et al.
Rheumatology (Oxford) 2006;45:1331-1337.
ABSTRACT | FULL TEXT  

Statins and Cancer Risk: A Literature-Based Meta-Analysis and Meta-Regression Analysis of 35 Randomized Controlled Trials
Bonovas et al.
JCO 2006;24:4808-4817.
ABSTRACT | FULL TEXT  

Treatments for Later-Life Depressive Conditions: A Meta-Analytic Comparison of Pharmacotherapy and Psychotherapy
Pinquart et al.
Am. J. Psychiatry 2006;163:1493-1501.
ABSTRACT | FULL TEXT  

A background to acupuncture and its use in chronic painful musculoskeletal conditions
White
The Journal of the Royal Society for the Promotion of Health 2006;126:219-227.
ABSTRACT  

Preventing pressure ulcers: a systematic review.
Reddy et al.
JAMA 2006;296:974-984.
ABSTRACT | FULL TEXT  

Segmental stabilizing exercises and low back pain. What is the evidence? A systematic review of randomized controlled trials
Rackwitz et al.
Clin Rehabil 2006;20:553-567.
ABSTRACT  

Weighing the evidence to formulate dietary guidelines.
Woolf
J. Am. Coll. Nutr. 2006;25:277S-284S.
ABSTRACT | FULL TEXT  

Effects of early treatment with statins on short-term clinical outcomes in acute coronary syndromes: a meta-analysis of randomized controlled trials.
Briel et al.
JAMA 2006;295:2046-2056.
ABSTRACT | FULL TEXT  

Body Mass Index and Adenocarcinomas of the Esophagus or Gastric Cardia: A Systematic Review and Meta-analysis.
Kubo and Corley
Cancer Epidemiol. Biomarkers Prev. 2006;15:872-878.
ABSTRACT | FULL TEXT  

Depression and Risk for Alzheimer Disease: Systematic Review, Meta-analysis, and Metaregression Analysis.
Ownby et al.
Arch Gen Psychiatry 2006;63:530-538.
ABSTRACT | FULL TEXT  

Angiotensin-Converting Enzyme Inhibitors in Patients With Coronary Artery Disease and Absence of Heart Failure or Left Ventricular Systolic Dysfunction: An Overview of Long-term Randomized Controlled Trials.
Danchin et al.
Arch Intern Med 2006;166:787-796.
ABSTRACT | FULL TEXT  

Bias in Clinical Intervention Research
Gluud
Am J Epidemiol 2006;163:493-501.
ABSTRACT | FULL TEXT  

Insulin Therapy and In-Hospital Mortality in Critically Ill Patients: Systematic Review and Meta-analysis of Randomized Controlled Trials
Pittas et al.
JPEN J Parenter Enteral Nutr 2006;30:164-172.
ABSTRACT | FULL TEXT  

Effects of Low-Carbohydrate vs Low-Fat Diets on Weight Loss and Cardiovascular Risk Factors: A Meta-analysis of Randomized Controlled Trials.
Nordmann et al.
Arch Intern Med 2006;166:285-293.
ABSTRACT | FULL TEXT  

Excess risk of fatal coronary heart disease associated with diabetes in men and women: meta-analysis of 37 prospective cohort studies
Huxley et al.
BMJ 2006;332:73-78.
ABSTRACT | FULL TEXT  

Use of Statins and Breast Cancer: A Meta-Analysis of Seven Randomized Clinical Trials and Nine Observational Studies
Bonovas et al.
JCO 2005;23:8606-8612.
ABSTRACT | FULL TEXT  

Amiodarone Prophylaxis Reduces Major Cardiovascular Morbidity and Length of Stay after Cardiac Surgery: A Meta-Analysis
Aasbo et al.
ANN INTERN MED 2005;143:327-336.
ABSTRACT | FULL TEXT  

Sirolimus-Eluting Stents vs Paclitaxel-Eluting Stents in Patients With Coronary Artery Disease: Meta-analysis of Randomized Trials
Kastrati et al.
JAMA 2005;294:819-825.
ABSTRACT | FULL TEXT  

Duration of Anticoagulation Following Venous Thromboembolism: A Meta-analysis
Ost et al.
JAMA 2005;294:706-715.
ABSTRACT | FULL TEXT  

A Systematic Review of the Risk Factors for Cervical Artery Dissection
Rubinstein et al.
Stroke 2005;36:1575-1580.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews: Synthesis of Topics Related to the Delivery, Organization, and Financing of Health Care
Bravata et al.
ANN INTERN MED 2005;142:1056-1065.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews of Educational Intervention Studies
Reed et al.
ANN INTERN MED 2005;142:1080-1089.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews That Assess Treatment Harms
Chou and Helfand
ANN INTERN MED 2005;142:1090-1099.
ABSTRACT | FULL TEXT  

Challenges in Using Nonrandomized Studies in Systematic Reviews of Treatment Interventions
Norris and Atkins
ANN INTERN MED 2005;142:1112-1119.
ABSTRACT | FULL TEXT  

Progestational Agents to Prevent Preterm Birth: A Meta-Analysis of Randomized Controlled Trials
Sanchez-Ramos and Kaunitz
Obstet Gynecol 2005;105:1484-1484.
FULL TEXT  

Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study
Moja et al.
BMJ 2005;330:1053.
ABSTRACT | FULL TEXT  

Systematic Review of Bariatric Surgery--Reply
Buchwald et al.
JAMA 2005;293:1726-1726.
FULL TEXT  

Intra-articular hyaluronic acid for the treatment of osteoarthritis of the knee: systematic review and meta-analysis
Arrich et al.
CMAJ 2005;172:1039-1043.
ABSTRACT | FULL TEXT  

Effect of Different Antilipidemic Agents and Diets on Mortality: A Systematic Review
Studer et al.
Arch Intern Med 2005;165:725-730.
ABSTRACT | FULL TEXT  

Evidence-based toxicology: a comprehensive framework for causation
Guzelian et al.
Hum Exp Toxicol 2005;24:161-201.
ABSTRACT  

Depression as a Risk Factor for Mortality in Patients With Coronary Heart Disease: A Meta-analysis
Barth et al.
Psychosom. Med. 2004;66:802-813.
ABSTRACT | FULL TEXT  

Interventions to Improve Research Participants' Understanding in Informed Consent for Research: A Systematic Review
Flory and Emanuel
JAMA 2004;292:1593-1601.
ABSTRACT | FULL TEXT  

Meta-Analysis of Interventions for Medication Adherence to Antihypertensives
Takiya et al.
The Annals of Pharmacotherapy 2004;38:1617-1624.
ABSTRACT | FULL TEXT  

Preoperative chemoradiotherapy for oesophageal cancer: a systematic review and meta-analysis
Fiorica et al.
Gut 2004;53:925-930.
ABSTRACT | FULL TEXT  

Challenges for Evidence-Based Physical Therapy: Accessing and Interpreting High-Quality Evidence on Therapy
Maher et al.
ptjournal 2004;84:644-654.
ABSTRACT | FULL TEXT  

Meta-Analysis of 2-Treatment Clinical Trials Including Both Continuous and Dichotomous Results
Babbs
Med Decis Making 2004;24:299-312.
ABSTRACT  

Mesothelioma and Lung Cancer Among Motor Vehicle Mechanics: a Meta-analysis
GOODMAN et al.
ANN OCCUP HYG 2004;48:309-326.
ABSTRACT | FULL TEXT  

Meta-Analysis: Respiratory Tolerance to Regular {beta}2-Agonist Use in Patients with Asthma
Salpeter et al.
ANN INTERN MED 2004;140:802-813.
ABSTRACT | FULL TEXT  

Interventions for the prevention of falls in older adults: systematic review and meta-analysis of randomised clinical trials
Chang et al.
BMJ 2004;328:680.
ABSTRACT | FULL TEXT  

Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials
Bhandari et al.
CMAJ 2004;170:477-480.
ABSTRACT | FULL TEXT  

Assessing the quality of research
Glasziou et al.
BMJ 2004;328:39-41.
FULL TEXT  

Prophylactic respiratory physiotherapy after cardiac surgery: systematic review
Pasquina et al.
BMJ 2003;327:1379.
ABSTRACT | FULL TEXT  

Clinical trials in orthodontics II: assessment of the quality of reporting of clinical trials published in three orthodontic journals between 1989 and 1998
Harrison
J. Orthod. 2003;30:309-315.
ABSTRACT | FULL TEXT  

Acyclovir Prophylaxis to Prevent Herpes Simplex Virus Recurrence at Delivery: A Systematic Review
Sheffield et al.
Obstet Gynecol 2003;102:1396-1403.
ABSTRACT | FULL TEXT  

Evidence Assessment of the Accuracy of Methods of Diagnosing Middle Ear Effusion in Children With Otitis Media With Effusion
Takata et al.
Pediatrics 2003;112:1379-1387.
ABSTRACT | FULL TEXT  

Anti-{beta}2-glycoprotein I, antiprothrombin antibodies, and the risk of thrombosis in the antiphospholipid syndrome
Galli et al.
Blood 2003;102:2717-2723.
FULL TEXT  

Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials
Maher et al.
ptjournal 2003;83:713-721.
ABSTRACT | FULL TEXT  

Effects of Computerized Physician Order Entry and Clinical Decision Support Systems on Medication Safety: A Systematic Review
Kaushal et al.
Arch Intern Med 2003;163:1409-1416.
ABSTRACT | FULL TEXT  

How to ... interpret the orthodontic literature
Glenny and Harrison
J. Orthod. 2003;30:159-164.
FULL TEXT  

Pharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin et al.
BMJ 2003;326:1167-1170.
ABSTRACT | FULL TEXT  

Methodological Quality Standards for Evaluation Research
Farrington
The ANNALS of the American Academy of Political and Social Science 2003;587:49-68.
ABSTRACT  

A meta-analysis of randomized controlled trials comparing coronary artery bypass graft with percutaneous transluminal coronary angioplasty: one- to eight-year outcomes
Hoffman et al.
J Am Coll Cardiol 2003;41:1293-1304.
ABSTRACT | FULL TEXT  

Assessing the Methodological Quality of Nonrandomized Intervention Studies
Saunders et al.
West J Nurs Res 2003;25:223-237.
ABSTRACT  

Complications of Evaluating the Scientific Quality of Randomized Clinical Trials
Pietrobon et al.
JBJS 2002;84:2307-2308.
FULL TEXT  

Quality of RCTs in Periodontology-- A Systematic Review
Montenegro et al.
J. Dent. Res. 2002;81:866-870.
ABSTRACT | FULL TEXT