This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (101)  • Contact me when this article is cited  Related Content  • Related letter  • Similar articles in JAMA  Topic Collections  • Journalology/ Peer Review/ Authorship  • Quality of Care  • Evidence-Based Medicine  • Statistics and Research Methods  • Alert me on articles by topic  Social Bookmarking   What's this?

John P. A. Ioannidis, MD; Joseph C. Cappelleri, PhD, MPH; Joseph Lau, MD

JAMA. 1998;279:1089-1093.

Context.— The extent of concordance between meta-analyses and large trials on the same topic has been investigated with different protocols. Inconsistent conclusions created confusion regarding the validity of these major tools of clinical evidence.

Objective.— To evaluate protocols comparing meta-analyses and large trials in order to understand if and why they disagree on the concordance of these 2 clinical research methods.

Design.— Systematic comparison of protocol designs, study selection, definitions of agreement, analysis methods, and reported discrepancies between large trials and meta-analyses.

Results.— More discrepancies were claimed when large trials were selected from influential journals (which may prefer trials disagreeing with prior evidence) than from already performed meta-analyses (which may target homogeneous trials) and when both primary and secondary (rather than only primary) end points were considered. Depending on how agreement was defined, kappa coefficients varied from 0.22 (low agreement) to 0.72 (excellent agreement). The correlation of treatment effects between large trials and meta-analyses varied from -0.12 to 0.76, but was more similar (0.50-0.76) when only primary end points were considered. When both the magnitude and uncertainty of treatment effects were considered, large trials disagreed with meta-analyses 10% to 23% of the time. Discrepancies were attributed to different disease risks, variable protocols, quality, and publication bias.

Conclusions.— Comparisons of large trials with meta-analyses may reach different conclusions depending on how trials and meta-analyses are selected and how end points and agreement are defined. Scrutiny of these 2 major research methods can enhance our appreciation of both for guiding medical practice.

From the National Institute of Allergy and Infectious Diseases, National Institutes of Health, Bethesda, Md (Dr Ioannidis); the Department of Clinical Research, Pfizer Central Research, Groton, Conn (Dr Cappelleri); and the Division of Clinical Care Research, New England Medical Center Hospitals, Tufts University School of Medicine, Boston, Mass (Dr Lau).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

RELATED LETTER

Comparing Results From Meta-analyses vs Large Trials
Jacques LeLorier, Genevieve Gregoire, John P. A. Ioannidis, Joseph C. Cappelleri, and Joseph Lau
JAMA. 1998;280(6):518-519.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Treatment of Bell Palsy: Translating Uncertainty Into Practice
Steiner
JAMA 2009;302:1003-1004.
FULL TEXT  

Effect of Folic Acid and B Vitamins on Risk of Cardiovascular Events and Total Mortality Among Women at High Risk for Cardiovascular Disease: A Randomized Trial
Albert et al.
JAMA 2008;299:2027-2036.
ABSTRACT | FULL TEXT  

Evidence from crossover trials: empirical evaluation and comparison against parallel arm trials
Lathyris et al.
Int J Epidemiol 2007;36:422-430.
ABSTRACT | FULL TEXT  

Meta-analysis of minimally invasive internal thoracic artery bypass versus percutaneous revascularisation for isolated lesions of the left anterior descending artery
Aziz et al.
BMJ 2007;334:617-617.
ABSTRACT | FULL TEXT  

Randomized Trial of Pulsed Corticosteroid Therapy for Primary Treatment of Kawasaki Disease
Newburger et al.
NEJM 2007;356:663-675.
ABSTRACT | FULL TEXT  

Clinical trial registration at Tobacco Control
Bero
Tobacco Control 2006;15:417-418.
FULL TEXT  

Bias in Clinical Intervention Research
Gluud
Am J Epidemiol 2006;163:493-501.
ABSTRACT | FULL TEXT  

Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies
Papanikolaou et al.
CMAJ 2006;174:635-641.
ABSTRACT | FULL TEXT  

A Comparison of Regional Versus General Anesthesia for Ambulatory Anesthesia: A Meta-Analysis of Randomized Controlled Trials
Liu et al.
Anesth. Analg. 2005;101:1634-1642.
ABSTRACT | FULL TEXT  

Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis
JAMA 2005;294:218-228.
ABSTRACT | FULL TEXT  

Why industry should register and disclose results of clinical studies--perspective of a recovering academic
Berlin
BMJ 2005;330:959-959.
FULL TEXT  

Comparison of Large Versus Smaller Randomized Trials for Mental Health-Related Interventions
Contopoulos-Ioannidis et al.
Am. J. Psychiatry 2005;162:578-584.
ABSTRACT | FULL TEXT  

Effect of Bisphosphonates on Periprosthetic Bone Mineral Density After Total Joint Arthroplasty. A Meta-Analysis
Bhandari et al.
JBJS 2005;87:293-301.
ABSTRACT | FULL TEXT  

Evidence-based Colloid Use in the Critically Ill: American Thoracic Society Consensus Statement
Am. J. Respir. Crit. Care Med. 2004;170:1247-1259.
FULL TEXT  

Meta-analyses and megatrials: neither is the infallible, universal standard
Furukawa
Evid. Based Ment. Health 2004;7:34-35.
FULL TEXT  

Blood-Borne Infections and Persons With Mental Illness: The Five-Site Health and Risk Study of Blood-Borne Infections Among Persons With Severe Mental Illness
Rosenberg et al.
Psychiatr. Serv. 2003;54:827-835.
ABSTRACT | FULL TEXT  

Cochrane Corner
Hachinski
Stroke 2002;33:1739-1739.
FULL TEXT  

A Quantitative, Systematic Review of Randomized Controlled Trials of Ephedrine Versus Phenylephrine for the Management of Hypotension During Spinal Anesthesia for Cesarean Delivery
Lee et al.
Anesth. Analg. 2002;94:920-926.
ABSTRACT | FULL TEXT  

Using Meta-analysis to Answer Clinical Questions
Laheij et al.
JAMA 2001;286:2669-2670.
FULL TEXT  

Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses
Kjaergard et al.
ANN INTERN MED 2001;135:982-989.
ABSTRACT | FULL TEXT  

Meta-analysis of randomized controlled trials on the comparative efficacy and safety of azithromycin against other antibiotics for lower respiratory tract infections
Contopoulos-Ioannidis et al.
J Antimicrob Chemother 2001;48:691-703.
ABSTRACT | FULL TEXT  

The Way in Which Intervention Studies Have "Personality" and why it is Important to Meta-Analysis
Lipsey and Wilson
Eval Health Prof 2001;24:236-254.
ABSTRACT  

Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Ioannidis et al.
JAMA 2001;286:821-830.
ABSTRACT | FULL TEXT  

ACE Gene Polymorphism in Cardiovascular Disease : Meta-Analyses of Small and Large Studies in Whites
Agerholm-Larsen et al.
Arterioscler. Thromb. Vasc. Bio. 2000;20:484-492.
ABSTRACT | FULL TEXT  

Resolving Discrepancies Between a Meta-analysis and a Subsequent Large Controlled Trial
DerSimonian and Levine
JAMA 1999;282:664-670.
ABSTRACT | FULL TEXT  

The Role of Meta-analysis in the Regulatory Process for Foods, Drugs, and Devices
Berlin and Colditz
JAMA 1999;281:830-834.
ABSTRACT | FULL TEXT  

Efficacy of Antiseptic-Impregnated Central Venous Catheters in Preventing Catheter-Related Bloodstream Infection: A Meta-analysis
Veenstra et al.
JAMA 1999;281:261-267.
ABSTRACT | FULL TEXT  

Evidence-Based Medicine in a Nutshell: A Guide to Finding and Using the Best Evidence in Caring for Patients
Bigby
Arch Dermatol 1998;134:1609-1618.
ABSTRACT | FULL TEXT  

Comparing Results From Meta-analyses vs Large Trials
LeLorier et al.
JAMA 1998;280:518-519.
FULL TEXT  

Evolution of treatment effects over time: Empirical insight from recursive cumulative metaanalyses
Ioannidis and Lau
Proc. Natl. Acad. Sci. USA 2001;98:831-836.
ABSTRACT | FULL TEXT