This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (499)  • Contact me when this article is cited  Related Content  • Related letters  • Related articles  • Similar articles in JAMA  Topic Collections  • Quality of Care  • Evidence-Based Medicine  • Quality of Care, Other  • Statistics and Research Methods  • Randomized Controlled Trial  • Alert me on articles by topic  Social Bookmarking   What's this?

Peter Jüni, MD; Anne Witschi, MD; Ralph Bloch, MD, PhD; Matthias Egger, MD, MSc

JAMA. 1999;282:1054-1060.

Context  Although it is widely recommended that clinical trials undergo some type of quality review, the number and variety of quality assessment scales that exist make it unclear how to achieve the best assessment.

Objective  To determine whether the type of quality assessment scale used affects the conclusions of meta-analytic studies.

Design and Setting  Meta-analysis of 17 trials comparing low-molecular-weight heparin (LMWH) with standard heparin for prevention of postoperative thrombosis using 25 different scales to identify high-quality trials. The association between treatment effect and summary scores and the association with 3 key domains (concealment of treatment allocation, blinding of outcome assessment, and handling of withdrawals) were examined in regression models.

Main Outcome Measure  Pooled relative risks of deep vein thrombosis with LMWH vs standard heparin in high-quality vs low-quality trials as determined by 25 quality scales.

Results  Pooled relative risks from high-quality trials ranged from 0.63 (95% confidence interval [CI], 0.44-0.90) to 0.90 (95% CI, 0.67-1.21) vs 0.52 (95% CI, 0.24-1.09) to 1.13 (95% CI, 0.70-1.82) for low-quality trials. For 6 scales, relative risks of high-quality trials were close to unity, indicating that LMWH was not significantly superior to standard heparin, whereas low-quality trials showed better protection with LMWH (P<.05). Seven scales showed the opposite: high quality trials showed an effect whereas low quality trials did not. For the remaining 12 scales, effect estimates were similar in the 2 quality strata. In regression analysis, summary quality scores were not significantly associated with treatment effects. There was no significant association of treatment effects with allocation concealment and handling of withdrawals. Open outcome assessment, however, influenced effect size with the effect of LMWH, on average, being exaggerated by 35% (95% CI, 1%-57%; P=.046).

Conclusions  Our data indicate that the use of summary scores to identify trials of high quality is problematic. Relevant methodological aspects should be assessed individually and their influence on effect sizes explored.

Author Affiliations: Clinical Epidemiology Study Group, Berne, Switzerland (Drs Jüni and Witschi), Institute for Medical Education, University of Berne (Drs Bloch and Jüni); and MRC Health Services Research Collaboration, Department of Social Medicine, University of Bristol, Bristol, England (Dr Egger).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

RELATED LETTERS

Scoring the Quality of Clinical Trials
Willem J. J. Assendelft, Bart W. Koes, Maurits W. van Tulder, Lex M. Bouter, Gerben ter Riet, Pieter Leffers, Maurice Zeegers, Jan P. Vandenbroucke, Peter Jüni, and Matthias Egger
JAMA. 2000;283(11):1421-1423.
EXTRACT | FULL TEXT  

The Ethics of Underpowered Clinical Trials
Janine E. Janosky, John R. Hughes, Richard J. Lilford, Scott D. Halpern, Jason H. T. Karlawish, and Jesse A. Berlin
JAMA. 2002;288(17):2118-2119.
EXTRACT | FULL TEXT  

Allocation Concealment in Clinical Trials
Kenneth F. Schulz, Douglas G. Altman, David Moher, Peter Jüni, Matthias Egger, Ethan M. Balk, Peter A. L. Bonis, Joseph Lau, and John P. A. Ioannidis
JAMA. 2002;288(19):2406-2409.
EXTRACT | FULL TEXT  

Prematurity and Later Cognitive Outcomes
Lex W. Doyle, Saroj Saigal, David L. Streiner, Adnan T. Bhutta, Mario Cleves, Patrick H. Casey, and K. J. S. Anand
JAMA. 2002;288(20):2542-2543.
EXTRACT | FULL TEXT  

RELATED ARTICLES

Empirical Evidence of Design-Related Bias in Studies of Diagnostic Tests
Jeroen G. Lijmer, Ben Willem Mol, Siem Heisterkamp, Gouke J. Bonsel, Martin H. Prins, Jan H. P. van der Meulen, and Patrick M. M. Bossuyt
JAMA. 1999;282(11):1061-1066.
ABSTRACT | FULL TEXT  

Measuring the Quality of Trials: The Quality of Quality Scales
Jesse A. Berlin and Drummond Rennie
JAMA. 1999;282(11):1083-1085.
EXTRACT | FULL TEXT  

September 15, 1999
JAMA. 1999;282(11):1099-1100.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Quality of randomised trials in COPD
Bausch et al.
Eur Respir J 2009;34:1060-1065.
ABSTRACT | FULL TEXT  

Reporting Experiments in Homeopathic Basic Research--Description of the Checklist Development
Stock-Schroer et al.
Evid Based Complement Alternat Med 2009;0:nep170v1-nep170.
ABSTRACT | FULL TEXT  

Goal-directed haemodynamic therapy and gastrointestinal complications in major surgery: a meta-analysis of randomized controlled trials
Giglio et al.
Br J Anaesth 2009;103:637-646.
ABSTRACT | FULL TEXT  

Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling et al.
BMJ 2009;339:b4012-b4012.
ABSTRACT | FULL TEXT  

The Principles of Effective Correctional Treatment Also Apply To Sexual Offenders: A Meta-Analysis
Hanson et al.
Criminal Justice and Behavior 2009;36:865-891.
ABSTRACT  

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration
Liberati et al.
ANN INTERN MED 2009;151:W-65-W-94.
ABSTRACT | FULL TEXT  

Sexual Abuse and Lifetime Diagnosis of Somatic Disorders: A Systematic Review and Meta-analysis
Paras et al.
JAMA 2009;302:550-561.
ABSTRACT | FULL TEXT  

Meta-analysis: Travel and Risk for Venous Thromboembolism
Chandra et al.
ANN INTERN MED 2009;151:180-190.
ABSTRACT | FULL TEXT  

Hypercoagulable State in Cushing's Syndrome: A Systematic Review
Van Zaane et al.
J. Clin. Endocrinol. Metab. 2009;94:2743-2750.
ABSTRACT | FULL TEXT  

The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration
Liberati et al.
BMJ 2009;339:b2700-b2700.
ABSTRACT | FULL TEXT  

The Relative Renal Safety of Iodixanol Compared With Low-Osmolar Contrast Media: A Meta-Analysis of Randomized Controlled Trials
Reed et al.
J Am Coll Cardiol Intv 2009;2:645-654.
ABSTRACT | FULL TEXT  

Comparison of Primary Percutaneous Coronary Intervention and Fibrinolytic Therapy in ST-Segment-Elevation Myocardial Infarction: Bayesian Hierarchical Meta-Analyses of Randomized Controlled Trials and Observational Studies
Huynh et al.
Circulation 2009;119:3101-3109.
ABSTRACT | FULL TEXT  

Allogeneic Stem Cell Transplantation for Acute Myeloid Leukemia in First Complete Remission: Systematic Review and Meta-analysis of Prospective Clinical Trials
Koreth et al.
JAMA 2009;301:2349-2361.
ABSTRACT | FULL TEXT  

Does exercise increase the risk of upper respiratory tract infections?
Moreira et al.
Br Med Bull 2009;90:111-131.
ABSTRACT | FULL TEXT  

A Comparison of Abciximab and Small-Molecule Glycoprotein IIb/IIIa Inhibitors in Patients Undergoing Primary Percutaneous Coronary Intervention: A Meta-Analysis of Contemporary Randomized Controlled Trials
Gurm et al.
Circ Cardiovasc Interv 2009;2:230-236.
ABSTRACT | FULL TEXT  

Review: Methodological quality in medical evidence, quo vadis?
Diaz-Insua
Therapeutic Advances in Urology 2009;1:51-59.
ABSTRACT  

Association between change in high density lipoprotein cholesterol and cardiovascular disease morbidity and mortality: systematic review and meta-regression analysis
Briel et al.
BMJ 2009;338:b92-b92.
ABSTRACT | FULL TEXT  

Immunoglobulin Prophylaxis in Hematopoietic Stem Cell Transplantation: Systematic Review and Meta-Analysis
Raanani et al.
JCO 2009;27:770-781.
ABSTRACT | FULL TEXT  

Reviewing evidence on complex social interventions: appraising implementation in systematic reviews of the health effects of organisational-level workplace interventions
Egan et al.
J. Epidemiol. Community Health 2009;63:4-11.
ABSTRACT | FULL TEXT  

Screening Colonoscopies by Primary Care Physicians: A Meta-Analysis
Wilkins et al.
Ann Fam Med 2009;7:56-62.
ABSTRACT | FULL TEXT  

Study Quality Assessment in Systematic Reviews of Research on Intervention Effects
Wells and Littell
Research on Social Work Practice 2009;19:52-62.
ABSTRACT  

Interventions to lower the glycemic response to carbohydrate foods with a low-viscosity fiber (resistant maltodextrin): meta-analysis of randomized controlled trials
Livesey and Tagami
Am. J. Clin. Nutr. 2009;89:114-125.
ABSTRACT | FULL TEXT  

Acupuncture for the Management of Chronic Headache: A Systematic Review
Sun and Gan
Anesth. Analg. 2008;107:2038-2047.
ABSTRACT | FULL TEXT  

Amitriptyline in the treatment of fibromyalgia: a systematic review of its efficacy
Nishishinya et al.
Rheumatology (Oxford) 2008;47:1741-1746.
ABSTRACT | FULL TEXT  

Primary Prevention of Cardiovascular Mortality and Events With Statin Treatments A Network Meta-Analysis Involving More Than 65,000 Patients.
Mills et al.
J Am Coll Cardiol 2008;52:1769-1781.
ABSTRACT | FULL TEXT  

Attitudes to walking and cycling among children, young people and parents: a systematic review
Lorenc et al.
J. Epidemiol. Community Health 2008;62:852-857.
ABSTRACT | FULL TEXT  

Quality of Reporting of Randomized Controlled Trials in General Endocrinology Literature
Rios et al.
J. Clin. Endocrinol. Metab. 2008;93:3810-3816.
ABSTRACT | FULL TEXT  

Overviews and Systematic Reviews on Low Back Pain
Chou et al.
ANN INTERN MED 2008;148:791-792.
FULL TEXT  

Cognitive Behavior Therapy for Schizophrenia: Effect Sizes, Clinical Models, and Methodological Rigor
Wykes et al.
Schizophr Bull 2008;34:523-537.
ABSTRACT | FULL TEXT  

Empirical Evidence of Bias in the Design of Experimental Stroke Studies: A Metaepidemiologic Approach
Crossley et al.
Stroke 2008;39:929-934.
ABSTRACT | FULL TEXT  

Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo et al.
ptjournal 2008;88:156-175.
ABSTRACT | FULL TEXT  

Perspectives on Evidence-Based Research in Education--What Works? Issues in Synthesizing Educational Program Evaluations
Slavin
EDUCATIONAL RESEARCHER 2008;37:5-14.
ABSTRACT | FULL TEXT  

Oral Drug Therapy for Multiple Neglected Tropical Diseases: A Systematic Review
Reddy et al.
JAMA 2007;298:1911-1924.
ABSTRACT | FULL TEXT  

A Meta-Analysis of 16 Randomized Trials of Sirolimus-Eluting Stents Versus Paclitaxel-Eluting Stents in Patients With Coronary Artery Disease
Schomig et al.
J Am Coll Cardiol 2007;50:1373-1380.
ABSTRACT | FULL TEXT  

The Regression Effect as a Neglected Source of Bias in Nonrandomized Intervention Trials and Systematic Reviews of Observational Studies
Weeks
Eval Health Prof 2007;30:254-265.
ABSTRACT  

Meta-analysis of APOE genotype and subarachnoid hemorrhage: Clinical outcome and delayed ischemia
Lanterna et al.
Neurology 2007;69:766-775.
ABSTRACT | FULL TEXT  

Statins and the Risk of Colorectal Cancer: A Meta-Analysis of 18 Studies Involving More Than 1.5 Million Patients
Bonovas et al.
JCO 2007;25:3462-3468.
ABSTRACT | FULL TEXT  

Are you reading what we are reading? The effect of who interprets medical images on estimates of diagnostic test accuracy in systematic reviews
Brealey and Westwood
Br. J. Radiol. 2007;80:674-677.
ABSTRACT | FULL TEXT  

Thyroid Dysfunction and Effects on Coagulation and Fibrinolysis: A Systematic Review
Squizzato et al.
J. Clin. Endocrinol. Metab. 2007;92:2415-2420.
ABSTRACT | FULL TEXT  

A Systematic Review of the Performance Characteristics of Clinical Event Monitor Signals Used to Detect Adverse Drug Events in the Hospital Setting
Handler et al.
J. Am. Med. Inform. Assoc. 2007;14:451-458.
ABSTRACT | FULL TEXT  

Commentary: Assessing the quality of observational studies--or a lesson from Mars
von Elm
Int J Epidemiol 2007;0:dym112v1-2.
FULL TEXT  

Meta-analysis: Acupuncture for Osteoarthritis of the Knee
Manheimer et al.
ANN INTERN MED 2007;146:868-877.
ABSTRACT | FULL TEXT  

A collaborative effort to apply the evidence-based review process to the field of nutrition: challenges, benefits, and lessons learned
Balk et al.
Am. J. Clin. Nutr. 2007;85:1448-1456.
ABSTRACT | FULL TEXT  

Sensory re-education after nerve injury of the upper limb: a systematic review
Oud et al.
Clin Rehabil 2007;21:483-494.
ABSTRACT  

Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding
Hrobjartsson et al.
Int J Epidemiol 2007;36:654-663.
ABSTRACT | FULL TEXT  

Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
Sanderson et al.
Int J Epidemiol 2007;36:666-676.
ABSTRACT | FULL TEXT  

Meta-analysis: Chondroitin for Osteoarthritis of the Knee or Hip
Reichenbach et al.
ANN INTERN MED 2007;146:580-590.
ABSTRACT | FULL TEXT  

Health technology assessment in obstetrics and gynaecology. Part 1: an overview of the process
Abedin et al.
The Obstetrician and Gynaecologist 2007;9:109-115.
ABSTRACT | FULL TEXT  

Parenting programmes for preventing tobacco, alcohol or drugs misuse in children <18: a systematic review
Petrie et al.
Health Educ Res 2007;22:177-191.
ABSTRACT | FULL TEXT  

Eligibility Criteria of Randomized Controlled Trials Published in High-Impact General Medical Journals: A Systematic Sampling Review
Van Spall et al.
JAMA 2007;297:1233-1240.
ABSTRACT | FULL TEXT  

Reporting of Outcomes in Orthopaedic Randomized Trials: Does Blinding of Outcome Assessors Matter?
Poolman et al.
JBJS 2007;89:550-558.
ABSTRACT | FULL TEXT  

Does pravastatin promote cancer in elderly patients? A meta-analysis
Bonovas and Sitaras
CMAJ 2007;176:649-654.
ABSTRACT | FULL TEXT  

Meta-analysis: Anticoagulant Prophylaxis to Prevent Symptomatic Venous Thromboembolism in Hospitalized Medical Patients
Dentali et al.
ANN INTERN MED 2007;146:278-288.
ABSTRACT | FULL TEXT  

Quality of Reporting of Cancer Prognostic Marker Studies: Association With Reported Prognostic Effect
Kyzas et al.
JNCI J Natl Cancer Inst 2007;99:236-243.
ABSTRACT | FULL TEXT  

Sleep-Disordered Breathing and Blood Pressure in Children: A Meta-analysis
Zintzaras and Kaditis
Arch Pediatr Adolesc Med 2007;161:172-178.
ABSTRACT | FULL TEXT  

Impact of Adjustment for Quality on Results of Metaanalyses of Diagnostic Accuracy
Leeflang et al.
Clin. Chem. 2007;53:164-172.
ABSTRACT | FULL TEXT  

Incorporating Quality of Evidence into Decision Analytic Modeling
Braithwaite et al.
ANN INTERN MED 2007;146:133-141.
ABSTRACT | FULL TEXT  

Economic evaluations of single- versus double-embryo transfer in IVF
Fiddelers et al.
Hum Reprod Update 2007;13:5-13.
ABSTRACT | FULL TEXT  

Mortality in randomized controlled trials comparing drug-eluting vs. bare metal stents in coronary artery disease: a meta-analysis
Nordmann et al.
Eur Heart J 2006;27:2784-2814.
ABSTRACT | FULL TEXT  

A systematic review of cephalometric facial soft tissue changes with the Activator and Bionator appliances in Class II division 1 subjects
Flores-Mir and Major
Eur J Orthod 2006;28:586-593.
ABSTRACT | FULL TEXT  

Intravenous thrombolysis in stroke patients of >=80 versus <80 years of age--a systematic review across cohort studies.
Engelter et al.
Age Ageing 2006;35:572-580.
ABSTRACT | FULL TEXT  

Acupuncture for peripheral joint osteoarthritis: A systematic review and meta-analysis
Kwon et al.
Rheumatology (Oxford) 2006;45:1331-1337.
ABSTRACT | FULL TEXT  

Statins and Cancer Risk: A Literature-Based Meta-Analysis and Meta-Regression Analysis of 35 Randomized Controlled Trials
Bonovas et al.
JCO 2006;24:4808-4817.
ABSTRACT | FULL TEXT  

Treatments for Later-Life Depressive Conditions: A Meta-Analytic Comparison of Pharmacotherapy and Psychotherapy
Pinquart et al.
Am. J. Psychiatry 2006;163:1493-1501.
ABSTRACT | FULL TEXT  

A background to acupuncture and its use in chronic painful musculoskeletal conditions
White
The Journal of the Royal Society for the Promotion of Health 2006;126:219-227.
ABSTRACT  

Preventing pressure ulcers: a systematic review.
Reddy et al.
JAMA 2006;296:974-984.
ABSTRACT | FULL TEXT  

Segmental stabilizing exercises and low back pain. What is the evidence? A systematic review of randomized controlled trials
Rackwitz et al.
Clin Rehabil 2006;20:553-567.
ABSTRACT  

Weighing the evidence to formulate dietary guidelines.
Woolf
J. Am. Coll. Nutr. 2006;25:277S-284S.
ABSTRACT | FULL TEXT  

Effects of early treatment with statins on short-term clinical outcomes in acute coronary syndromes: a meta-analysis of randomized controlled trials.
Briel et al.
JAMA 2006;295:2046-2056.
ABSTRACT | FULL TEXT  

Body Mass Index and Adenocarcinomas of the Esophagus or Gastric Cardia: A Systematic Review and Meta-analysis.
Kubo and Corley
Cancer Epidemiol. Biomarkers Prev. 2006;15:872-878.
ABSTRACT | FULL TEXT  

Depression and Risk for Alzheimer Disease: Systematic Review, Meta-analysis, and Metaregression Analysis.
Ownby et al.
Arch Gen Psychiatry 2006;63:530-538.
ABSTRACT | FULL TEXT  

Angiotensin-Converting Enzyme Inhibitors in Patients With Coronary Artery Disease and Absence of Heart Failure or Left Ventricular Systolic Dysfunction: An Overview of Long-term Randomized Controlled Trials.
Danchin et al.
Arch Intern Med 2006;166:787-796.
ABSTRACT | FULL TEXT  

Bias in Clinical Intervention Research
Gluud
Am J Epidemiol 2006;163:493-501.
ABSTRACT | FULL TEXT  

Insulin Therapy and In-Hospital Mortality in Critically Ill Patients: Systematic Review and Meta-analysis of Randomized Controlled Trials
Pittas et al.
JPEN J Parenter Enteral Nutr 2006;30:164-172.
ABSTRACT | FULL TEXT  

Effects of Low-Carbohydrate vs Low-Fat Diets on Weight Loss and Cardiovascular Risk Factors: A Meta-analysis of Randomized Controlled Trials.
Nordmann et al.
Arch Intern Med 2006;166:285-293.
ABSTRACT | FULL TEXT  

Excess risk of fatal coronary heart disease associated with diabetes in men and women: meta-analysis of 37 prospective cohort studies
Huxley et al.
BMJ 2006;332:73-78.
ABSTRACT | FULL TEXT  

Use of Statins and Breast Cancer: A Meta-Analysis of Seven Randomized Clinical Trials and Nine Observational Studies
Bonovas et al.
JCO 2005;23:8606-8612.
ABSTRACT | FULL TEXT  

Amiodarone Prophylaxis Reduces Major Cardiovascular Morbidity and Length of Stay after Cardiac Surgery: A Meta-Analysis
Aasbo et al.
ANN INTERN MED 2005;143:327-336.
ABSTRACT | FULL TEXT  

Sirolimus-Eluting Stents vs Paclitaxel-Eluting Stents in Patients With Coronary Artery Disease: Meta-analysis of Randomized Trials
Kastrati et al.
JAMA 2005;294:819-825.
ABSTRACT | FULL TEXT  

Duration of Anticoagulation Following Venous Thromboembolism: A Meta-analysis
Ost et al.
JAMA 2005;294:706-715.
ABSTRACT | FULL TEXT  

A Systematic Review of the Risk Factors for Cervical Artery Dissection
Rubinstein et al.
Stroke 2005;36:1575-1580.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews: Synthesis of Topics Related to the Delivery, Organization, and Financing of Health Care
Bravata et al.
ANN INTERN MED 2005;142:1056-1065.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews of Educational Intervention Studies
Reed et al.
ANN INTERN MED 2005;142:1080-1089.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews That Assess Treatment Harms
Chou and Helfand
ANN INTERN MED 2005;142:1090-1099.
ABSTRACT | FULL TEXT  

Challenges in Using Nonrandomized Studies in Systematic Reviews of Treatment Interventions
Norris and Atkins
ANN INTERN MED 2005;142:1112-1119.
ABSTRACT | FULL TEXT  

Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study
Moja et al.
BMJ 2005;330:1053.
ABSTRACT | FULL TEXT  

Systematic Review of Bariatric Surgery--Reply
Buchwald et al.
JAMA 2005;293:1726-1726.
FULL TEXT  

Intra-articular hyaluronic acid for the treatment of osteoarthritis of the knee: systematic review and meta-analysis
Arrich et al.
CMAJ 2005;172:1039-1043.
ABSTRACT | FULL TEXT  

Effect of Different Antilipidemic Agents and Diets on Mortality: A Systematic Review
Studer et al.
Arch Intern Med 2005;165:725-730.
ABSTRACT | FULL TEXT