This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on ISI (176)  • Contact me when this article is cited  Related Content  • Related letter  • Related article  • Similar articles in JAMA  Topic Collections  • Statistics and Research Methods  • Review  • Randomized Controlled Trial  • Alert me on articles by topic  Social Bookmarking   What's this?

John P. A. Ioannidis, MD; Anna-Bettina Haidich, MSc; Maroudia Pappa, MSc; Nikos Pantazis, MSc; Styliani I. Kokori, MD; Maria G. Tektonidou, MD; Despina G. Contopoulos-Ioannidis, MD; Joseph Lau, MD

JAMA. 2001;286:821-830.

Context  There is substantial debate about whether the results of nonrandomized studies are consistent with the results of randomized controlled trials on the same topic.

Objectives  To compare results of randomized and nonrandomized studies that evaluated medical interventions and to examine characteristics that may explain discrepancies between randomized and nonrandomized studies.

Data Sources  MEDLINE (1966–March 2000), the Cochrane Library (Issue 3, 2000), and major journals were searched.

Study Selection  Forty-five diverse topics were identified for which both randomized trials (n = 240) and nonrandomized studies (n = 168) had been performed and had been considered in meta-analyses of binary outcomes.

Data Extraction  Data on events per patient in each study arm and design and characteristics of each study considered in each meta-analysis were extracted and synthesized separately for randomized and nonrandomized studies.

Data Synthesis  Very good correlation was observed between the summary odds ratios of randomized and nonrandomized studies (r = 0.75; P<.001); however, nonrandomized studies tended to show larger treatment effects (28 vs 11; P = .009). Between-study heterogeneity was frequent among randomized trials alone (23%) and very frequent among nonrandomized studies alone (41%). The summary results of the 2 types of designs differed beyond chance in 7 cases (16%). Discrepancies beyond chance were less common when only prospective studies were considered (8%). Occasional differences in sample size and timing of publication were also noted between discrepant randomized and nonrandomized studies. In 28 cases (62%), the natural logarithm of the odds ratio differed by at least 50%, and in 15 cases (33%), the odds ratio varied at least 2-fold between nonrandomized studies and randomized trials.

Conclusions  Despite good correlation between randomized trials and nonrandomized studies—in particular, prospective studies—discrepancies beyond chance do occur and differences in estimated magnitude of treatment effect are very common.

Author Affiliations: Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina (Drs Ioannidis and Contopoulos-Ioannidis, and Ms Haidich), Department of Hygiene and Epidemiology, University of Athens School of Medicine (Ms Pappa and Mr Pantazis) and Laikon General Hospital (Drs Kokori and Tektonidou), Athens, Greece; Department of Pediatrics, George Washington University School of Medicine, Washington, DC (Dr Contopoulos-Ioannidis); and Division of Clinical Care Research, Department of Medicine, Tufts University School of Medicine, Boston, Mass (Drs Ioannidis and Lau).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati     What's this?

RELATED LETTER

Using Meta-analysis to Answer Clinical Questions
Robert J. F. Laheij, Demetrios S. Theodoropoulos, Georgios A. Theodoropoulos, Donna L. Pecoraro, John P. A. Ioannidis, Despina G. Contopoulos-Ioannidis, and Joseph Lau
JAMA. 2001;286(21):2669-2670.
EXTRACT | FULL TEXT  

RELATED ARTICLE

August 15, 2001
JAMA. 2001;286(7):863-864.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Adapted Behavior Therapy for Persistently Depressed Primary Care Patients: An Open Trial
Uebelacker et al.
Behav Modif 2009;33:374-395.
ABSTRACT  

Design, Conduct, and Interpretation of Nonrandomized Orthopaedic Studies: A Practical Approach. (All) Evidence Matters
Bhandari et al.
JBJS 2009;91:1-1.
FULL TEXT  

Is Personality a Key Predictor of Missing Study Data? An Analysis From a Randomized Controlled Trial
Jerant et al.
Ann Fam Med 2009;7:148-156.
ABSTRACT | FULL TEXT  

Use of primary care electronic medical record database in drug efficacy research on cardiovascular outcomes: comparison of database and randomised controlled trial findings
Tannen et al.
BMJ 2009;338:b81-b81.
ABSTRACT | FULL TEXT  

Systematic differences in treatment effect estimates between propensity score methods and logistic regression
Martens et al.
Int J Epidemiol 2008;37:1142-1147.
ABSTRACT | FULL TEXT  

False-Positive Results in Cancer Epidemiology: A Plea for Epistemological Modesty
Boffetta et al.
JNCI J Natl Cancer Inst 2008;100:988-995.
ABSTRACT | FULL TEXT  

Carts, Horses, and Evidence
Koretz
JPEN J Parenter Enteral Nutr 2008;32:482-485.
FULL TEXT  

Reasons or excuses for avoiding meta-analysis in forest plots
Ioannidis et al.
BMJ 2008;336:1413-1415.
FULL TEXT  

Randomized Clinical Trials and Observational Studies: Guidelines for Assessing Respective Strengths and Limitations
Hannan
J Am Coll Cardiol Intv 2008;1:211-217.
ABSTRACT | FULL TEXT  

Persistence of Contradicted Claims in the Literature
Tatsioni et al.
JAMA 2007;298:2517-2526.
ABSTRACT | FULL TEXT  

Considerations of Study Design
Koretz
Nutr Clin Pract 2007;22:593-598.
ABSTRACT | FULL TEXT  

Should Meta-Analyses of Interventions Include Observational Studies in Addition to Randomized Controlled Trials? A Critical Examination of Underlying Principles
Shrier et al.
Am J Epidemiol 2007;166:1203-1209.
ABSTRACT | FULL TEXT  

Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy
Nicholl
J. Epidemiol. Community Health 2007;61:1010-1013.
ABSTRACT | FULL TEXT  

Healthy Steps for Young Children: Sustained Results at 5.5 Years
Minkovitz et al.
Pediatrics 2007;120:e658-e668.
ABSTRACT | FULL TEXT  

How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania et al.
ANN INTERN MED 2007;147:224-233.
ABSTRACT | FULL TEXT  

Effects of platelet glycoprotein IIb/IIIa receptor blockers in non-ST segment elevation acute coronary syndromes: benefit and harm in different age subgroups
Hernandez et al.
Heart 2007;93:450-455.
ABSTRACT | FULL TEXT  

Quality of Reporting of Cancer Prognostic Marker Studies: Association With Reported Prognostic Effect
Kyzas et al.
JNCI J Natl Cancer Inst 2007;99:236-243.
ABSTRACT | FULL TEXT  

Arthroscopic Compared with Open Repairs for Recurrent Anterior Shoulder Instability. A Systematic Review and Meta-Analysis of the Literature
Lenters et al.
JBJS 2007;89:244-254.
ABSTRACT | FULL TEXT  

Evidence Summaries and Synthesis: Necessary but Insufficient Approach for Determining Clinical Practice of Integrated Medicine?
Coulter
Integr Cancer Ther 2006;5:282-286.
ABSTRACT  

Con: Aprotinin Has a Good Efficacy and Safety Profile Relative to Other Alternatives for Prevention of Bleeding in Cardiac Surgery
Beattie and Karkouti
Anesth. Analg. 2006;103:1360-1364.
FULL TEXT  

Meta-analysis: risk for hypertension in living kidney donors.
Boudville et al.
ANN INTERN MED 2006;145:185-196.
ABSTRACT | FULL TEXT  

Effectiveness of antipsychotic treatments in a nationwide cohort of patients in community care after first hospitalisation due to schizophrenia and schizoaffective disorder: observational follow-up study
Tiihonen et al.
BMJ 2006;333:224.
ABSTRACT | FULL TEXT  

Mixed comparison of stroke prevention treatments in individuals with nonrheumatic atrial fibrillation.
Cooper et al.
Arch Intern Med 2006;166:1269-1275.
ABSTRACT | FULL TEXT  

Commentary: Grading the credibility of molecular evidence for complex diseases
Ioannidis
Int J Epidemiol 2006;35:572-578.
FULL TEXT  

Bias in Clinical Intervention Research
Gluud
Am J Epidemiol 2006;163:493-501.
ABSTRACT | FULL TEXT  

Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies
Papanikolaou et al.
CMAJ 2006;174:635-641.
ABSTRACT | FULL TEXT  

The Perspective Less Viewed
Koretz
JPEN J Parenter Enteral Nutr 2006;30:63-65.
FULL TEXT  

Obesity and Risk of New-Onset Atrial Fibrillation After Cardiac Surgery
Zacharias et al.
Circulation 2005;112:3247-3255.
ABSTRACT | FULL TEXT  

Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002
Gilbert et al.
Int J Epidemiol 2005;34:874-887.
ABSTRACT | FULL TEXT  

Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis
JAMA 2005;294:218-228.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews That Evaluate Drug Efficacy or Effectiveness
Santaguida et al.
ANN INTERN MED 2005;142:1066-1072.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews That Assess Treatment Harms
Chou and Helfand
ANN INTERN MED 2005;142:1090-1099.
ABSTRACT | FULL TEXT  

Challenges in Systematic Reviews of Therapeutic Devices and Procedures
Hartling et al.
ANN INTERN MED 2005;142:1100-1111.
ABSTRACT | FULL TEXT  

Diet and Cancer: The Disconnect Between Epidemiology and Randomized Clinical Trials
Meyskens and Szabo
Cancer Epidemiol. Biomarkers Prev. 2005;14:1366-1369.
ABSTRACT | FULL TEXT  

Effects of Obesity and Small Body Size on Operative and Long-Term Outcomes of Coronary Artery Bypass Surgery: A Propensity-Matched Analysis
Habib et al.
Ann. Thorac. Surg. 2005;79:1976-1986.
ABSTRACT | FULL TEXT  

Relative Citation Impact of Various Study Designs in the Health Sciences
Patsopoulos et al.
JAMA 2005;293:2362-2366.
ABSTRACT | FULL TEXT  

Statins decrease perioperative cardiac complications in patients undergoing noncardiac vascular surgery: The Statins for Risk Reduction in Surgery (StaRRS) study
O'Neil-Callahan et al.
J Am Coll Cardiol 2005;45:336-342.
ABSTRACT | FULL TEXT  

Making Policy When The Evidence Is In Dispute
Atkins et al.
Health Aff (Millwood) 2005;24:102-113.
ABSTRACT | FULL TEXT  

Potential Pitfalls of Disease-Specific Guidelines for Patients with Multiple Conditions
Tinetti et al.
NEJM 2004;351:2870-2874.
FULL TEXT  

Grading quality of evidence and strength of recommendations
BMJ 2004;328:1490.
ABSTRACT | FULL TEXT  

Letter to the editor
Zanolla
Eur J Heart Fail 2004;6:509-509.
FULL TEXT  

A Practice-Based Intervention to Enhance Quality of Care in the First 3 Years of Life: The Healthy Steps for Young Children Program
Minkovitz et al.
JAMA 2003;290:3081-3091.
ABSTRACT | FULL TEXT  

Rigorous uncertainty: why RA Fisher is important
Marks
Int J Epidemiol 2003;32:932-937.
FULL TEXT  

Hope for Perinatal Prevention of Cerebral Palsy
Tyson and Gilstrap
JAMA 2003;290:2730-2732.
FULL TEXT  

Efficacy of Postoperative Epidural Analgesia: A Meta-analysis
Block et al.
JAMA 2003;290:2455-2463.
ABSTRACT | FULL TEXT  

What Constitutes Optimal Care for Older Women With Breast Cancer?
Silliman
JCO 2003;21:3554-3556.
FULL TEXT  

Determinants of treatment strategies and survival in acute myocardial infarction: a population-based study in the Florence district, Italy: Results of the acute myocardial infarction Florence registry (AMI-Florence),
Buiatti et al.
Eur Heart J 2003;24:1195-1203.
ABSTRACT | FULL TEXT  

Review: observational studies adjusting for socioeconomic status and lifestyle show no association between HRT and CAD
Colditz
Evid. Based Med. 2003;8:46-46.
FULL TEXT  

Review: risks and benefits of HRT comparing various sources of evidence
Colditz
Evid. Based Med. 2003;8:47-47.
FULL TEXT  

Complementary or alternative medicine: the need for plausibility
Hoffer
CMAJ 2003;168:180-182.
FULL TEXT  

Risk of Hemolytic Uremic Syndrome From Antibiotic Treatment of Escherichia coli O157:H7 Colitis
Gill et al.
JAMA 2002;288:3110-3112.
FULL TEXT  

Evidence-Based Education Policies: Transforming Educational Practice and Research
Slavin
EDUCATIONAL RESEARCHER 2002;31:15-21.
ABSTRACT  

Placebos that harm: sham surgery controls in clinical trials
London and Kadane
Stat Methods Med Res 2002;11:413-427.
ABSTRACT  

Systematic reviews in epidemiology: why are we so far behind?
Dickersin
Int J Epidemiol 2002;31:6-12.
FULL TEXT  

Cost-effectiveness of Homocysteine-Lowering Therapy to Prevent Coronary Heart Disease
Bostom et al.
JAMA 2002;287:190-192.
FULL TEXT  

Using Meta-analysis to Answer Clinical Questions
Laheij et al.
JAMA 2001;286:2669-2670.
FULL TEXT