This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on ISI (201)  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Topic Collections  • Journalology/ Peer Review/ Authorship  • Statistics and Research Methods  • Randomized Controlled Trial  • Prognosis/ Outcomes  • Alert me on articles by topic

Comparison of Protocols to Published Articles

An-Wen Chan, MD, DPhil; Asbjørn Hróbjartsson, MD, PhD; Mette T. Haahr, BSc; Peter C. Gøtzsche, MD, DrMedSci; Douglas G. Altman, DSc

JAMA. 2004;291:2457-2465.

Context  Selective reporting of outcomes within published studies based on the nature or direction of their results has been widely suspected, but direct evidence of such bias is currently limited to case reports.

Objective  To study empirically the extent and nature of outcome reporting bias in a cohort of randomized trials.

Design  Cohort study using protocols and published reports of randomized trials approved by the Scientific-Ethical Committees for Copenhagen and Frederiksberg, Denmark, in 1994-1995. The number and characteristics of reported and unreported trial outcomes were recorded from protocols, journal articles, and a survey of trialists. An outcome was considered incompletely reported if insufficient data were presented in the published articles for meta-analysis. Odds ratios relating the completeness of outcome reporting to statistical significance were calculated for each trial and then pooled to provide an overall estimate of bias. Protocols and published articles were also compared to identify discrepancies in primary outcomes.

Main Outcome Measures  Completeness of reporting of efficacy and harm outcomes and of statistically significant vs nonsignificant outcomes; consistency between primary outcomes defined in the most recent protocols and those defined in published articles.

Results  One hundred two trials with 122 published journal articles and 3736 outcomes were identified. Overall, 50% of efficacy and 65% of harm outcomes per trial were incompletely reported. Statistically significant outcomes had a higher odds of being fully reported compared with nonsignificant outcomes for both efficacy (pooled odds ratio, 2.4; 95% confidence interval [CI], 1.4-4.0) and harm (pooled odds ratio, 4.7; 95% CI, 1.8-12.0) data. In comparing published articles with protocols, 62% of trials had at least 1 primary outcome that was changed, introduced, or omitted. Eighty-six percent of survey responders (42/49) denied the existence of unreported outcomes despite clear evidence to the contrary.

Conclusions  The reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols. Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention. To ensure transparency, planned trials should be registered and protocols should be made publicly available prior to trial completion.

Author Affiliations: Centre for Statistics in Medicine, Institute of Health Sciences, Oxford, England (Drs Chan and Altman), Nordic Cochrane Centre, Copenhagen, Denmark (Drs Hróbjartsson and Gøtzsche and Ms Haahr), University Health Network, University of Toronto, Toronto, Ontario (Dr Chan).

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?
Thorlund et al.
Int J Epidemiol 2008;0:dyn179v1-dyn179.
ABSTRACT | FULL TEXT  

Apparently conclusive meta-analyses may be inconclusive - Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses
Brok et al.
Int J Epidemiol 2008;0:dyn188v1-dyn188.
ABSTRACT | FULL TEXT  

Evaluation of the Potential Excess of Statistically Significant Findings in Published Genetic Association Studies: Application to Alzheimer's Disease
Kavvoura et al.
Am J Epidemiol 2008;0:kwn206v1-kwn206.
ABSTRACT | FULL TEXT  

Commercialism, Loss of Professionalism, and the Effect on Journals
Liesegang
Arch Ophthalmol 2008;126:1292-1295.
FULL TEXT  

Reporting Bias in Diagnostic and Prognostic Studies: Time for Action
Rifai et al.
Clin. Chem. 2008;54:1101-1103.
FULL TEXT  

What is missing from descriptions of treatment in trials and reviews?
Glasziou et al.
BMJ 2008;336:1472-1474.
FULL TEXT  

Patient-Important Outcomes in Registered Diabetes Trials
Gandhi et al.
JAMA 2008;299:2543-2549.
ABSTRACT | FULL TEXT  

Research Reporting and Evidence of Effectiveness: Why "No Difference" Matters
Hackbarth
Am J Crit Care 2008;17:218-220.
FULL TEXT  

How Much of the Data Published in Observational Studies of the Association between Diet and Prostate or Bladder Cancer Is Usable for Meta-Analysis?
Bekkering et al.
Am J Epidemiol 2008;167:1017-1026.
ABSTRACT | FULL TEXT  

PROactive: time for a critical appraisal
Betteridge et al.
Eur Heart J 2008;29:969-983.
ABSTRACT | FULL TEXT  

Publishing study protocols
Johnston
Inj. Prev. 2008;14:73-73.
FULL TEXT  

Exploring the Geometry of Treatment Networks
Salanti et al.
ANN INTERN MED 2008;148:544-553.
ABSTRACT | FULL TEXT  

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
Wood et al.
BMJ 2008;336:601-605.
ABSTRACT | FULL TEXT  

A Case for Clarity, Consistency, and Helpfulness: State-of-the-Art Clinical Practice Guidelines in Endocrinology Using the Grading of Recommendations, Assessment, Development, and Evaluation System
Swiglo et al.
J. Clin. Endocrinol. Metab. 2008;93:666-673.
ABSTRACT | FULL TEXT  

Moving from opacity to transparency in pharmaceutical policy
Dhalla and Laupacis
CMAJ 2008;178:428-431.
FULL TEXT  

Improving access to research data in Europe
Mladovsky et al.
BMJ 2008;336:287-288.
FULL TEXT  

Adding aripiprazole improves major depressive disorder following incomplete response to antidepressants alone
Cipriani and Barbui
Evid. Based Ment. Health 2008;11:15-15.
FULL TEXT  

Assessment of cumulative evidence on genetic associations: interim guidelines
Ioannidis et al.
Int J Epidemiol 2008;37:120-132.
ABSTRACT | FULL TEXT  

Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy
Turner et al.
NEJM 2008;358:252-260.
ABSTRACT | FULL TEXT  

RE: "INVITED COMMENTARY: HORMONE THERAPY AND RISK OF CORONARY HEART DISEASE WHY RENEW THE FOCUS ON THE EARLY YEARS OF MENOPAUSE?"
Doren and Greiser
Am J Epidemiol 2007;166:1479-1480.
FULL TEXT  

State of the Evidence on Acute Asthma Management in Children: A Critical Appraisal of Systematic Reviews
Boluyt et al.
Pediatrics 2007;120:1334-1343.
ABSTRACT | FULL TEXT  

Increased Mortality, Postoperative Morbidity, and Cost After Red Blood Cell Transfusion in Patients Having Cardiac Surgery
Murphy et al.
Circulation 2007;116:2544-2552.
ABSTRACT | FULL TEXT  

Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration
Vandenbroucke et al.
ANN INTERN MED 2007;147:W-163-W-194.
ABSTRACT | FULL TEXT  

Claims of Sex Differences: An Empirical Assessment in Genetic Associations
Patsopoulos et al.
JAMA 2007;298:880-893.
ABSTRACT | FULL TEXT  

Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal et al.
Int J Epidemiol 2007;36:847-857.
ABSTRACT | FULL TEXT  

An exploratory test for an excess of significant findings
Ioannidis and Trikalinos
Clin Trials 2007;4:245-253.
ABSTRACT  

Authors' response to V Johnson and Y Yuan
Ioannidis and Trikalinos
Clin Trials 2007;4:256-257.
 

Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding
Hrobjartsson et al.
Int J Epidemiol 2007;36:654-663.
ABSTRACT | FULL TEXT  

Issues in the Registration of Clinical Trials
Zarin et al.
JAMA 2007;297:2112-2120.
ABSTRACT | FULL TEXT  

Ethical Considerations in the Interpretation and Communication of Clinical Trial Results
Coultas
Proc Am Thorac Soc 2007;4:194-199.
ABSTRACT | FULL TEXT  

Reporting research results: a moral obligation for all researchers/Presenter les resultants de recherche : une obligation morale pour tous les chercheurs
Moher
Canadian J. Anesthesia 2007;54:331-335.
FULL TEXT  

Genetic Effects versus Bias for Candidate Polymorphisms in Myocardial Infarction: Case Study and Overview of Large-Scale Evidence
Ntzani et al.
Am J Epidemiol 2007;165:973-984.
ABSTRACT | FULL TEXT  

Reporting of radiographic methods in randomised controlled trials assessing structural outcomes in rheumatoid arthritis
Baron et al.
Ann Rheum Dis 2007;66:651-657.
ABSTRACT | FULL TEXT  

Effects of platelet glycoprotein IIb/IIIa receptor blockers in non-ST segment elevation acute coronary syndromes: benefit and harm in different age subgroups
Hernandez et al.
Heart 2007;93:450-455.
ABSTRACT | FULL TEXT  

Oral presentation bias: a retrospective cohort study
Decullier and Chapuis
J. Epidemiol. Community Health 2007;61:190-193.
ABSTRACT | FULL TEXT  

Association Between Unreported Outcomes and Effect Size Estimates in Cochrane Meta-analyses
Furukawa et al.
JAMA 2007;297:468-470.
FULL TEXT  

Meta-analysis: Intravenous Immunoglobulin in Critically Ill Adult Patients with Sepsis
Turgeon et al.
ANN INTERN MED 2007;146:193-203.
ABSTRACT | FULL TEXT  

The Use of the World Wide Web by Medical Journals in 2003 and 2005: An Observational Study
Schriger et al.
Pediatrics 2007;119:e53-e60.
ABSTRACT | FULL TEXT  

'They would say that, wouldn't they?' A reader's guide to author and sponsor biases in clinical research
Bastian
JRSM 2006;99:611-614.
FULL TEXT  

Clinical trial registration at Tobacco Control
Bero
Tobacco Control 2006;15:417-418.
FULL TEXT  

Latency of the mfVEP to diagnose glaucoma?
Bach
Br. J. Ophthalmol. 2006;90:1076-1077.
FULL TEXT  

Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors
Haahr and Hrobjartsson
Clin Trials 2006;3:360-365.
ABSTRACT  

Believability of relative risks and odds ratios in abstracts: cross sectional study
Gotzsche
BMJ 2006;333:231-234.
ABSTRACT | FULL TEXT  

Commentary: Grading the credibility of molecular evidence for complex diseases
Ioannidis
Int J Epidemiol 2006;35:572-578.
FULL TEXT  

Reported outcomes in major cardiovascular clinical trials funded by for-profit and not-for-profit organizations: 2000-2005.
Ridker and Torres
JAMA 2006;295:2270-2274.
ABSTRACT | FULL TEXT  

Waiving confidentiality for the greater good.
Chan et al.
BMJ 2006;332:1086-1089.
FULL TEXT  

Constraints on Publication Rights in Industry-Initiated Clinical Trials
Gotzsche et al.
JAMA 2006;295:1645-1646.
FULL TEXT  

Bias in Clinical Intervention Research
Gluud
Am J Epidemiol 2006;163:493-501.
ABSTRACT | FULL TEXT  

Quality of Reporting of Noninferiority and Equivalence Randomized Trials
Le Henanff et al.
JAMA 2006;295:1147-1151.
ABSTRACT | FULL TEXT  

Lessons From and Cautions About Noninferiority and Equivalence Randomized Trials
Gotzsche
JAMA 2006;295:1172-1174.
FULL TEXT  

On bias and transparency in the development of influential recommendations
Laupacis
CMAJ 2006;174:335-336.
FULL TEXT  

A Call for Fairness in Formulary Decisions
Levinson and Laupacis
Arch Intern Med 2006;166:16-18.
FULL TEXT  

Registries and Registration of Clinical Trials
Haug et al.
NEJM 2005;353:2811-2812.
FULL TEXT  

Outcome selection bias in meta-analysis
Williamson et al.
Stat Methods Med Res 2005;14:515-524.
ABSTRACT  

Systematic review of the effects of n-3 fatty acids in inflammatory bowel disease
MacLean et al.
Am. J. Clin. Nutr. 2005;82:611-619.
ABSTRACT | FULL TEXT  

A Network of Investigator Networks in Human Genome Epidemiology
Ioannidis et al.
Am J Epidemiol 2005;162:302-304.
ABSTRACT | FULL TEXT  

Genomic Epidemiology of Complex Disease: The Need for an Electronic Evidence-based Approach to Research Synthesis
Bracken
Am J Epidemiol 2005;162:297-301.
ABSTRACT | FULL TEXT  

Hard Sell: The Evolution of a Viagra Salesman
Morgan
JAMA 2005;294:496-496.
FULL TEXT  

Identification of Clinically Useful Cancer Prognostic Factors: What Are We Missing?
McShane et al.
JNCI J Natl Cancer Inst 2005;97:1023-1025.
FULL TEXT  

Selective Reporting Biases in Cancer Prognostic Factor Studies
Kyzas et al.
JNCI J Natl Cancer Inst 2005;97:1043-1055.
ABSTRACT | FULL TEXT  

Fate of biomedical research protocols and publication bias in France: retrospective cohort study
Decullier et al.
BMJ 2005;331:19.
ABSTRACT | FULL TEXT  

Patellar Resurfacing in Total Knee Arthroplasty. A Meta-Analysis
Pakos et al.
JBJS 2005;87:1438-1445.
ABSTRACT | FULL TEXT  

Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study
Pildal et al.
BMJ 2005;330:1049.
ABSTRACT | FULL TEXT  

Expectant, Medical, or Surgical Management of First-Trimester Miscarriage: A Meta-Analysis
Sotiriadis et al.
Obstet Gynecol 2005;105:1104-1113.
ABSTRACT | FULL TEXT  

Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1)
Krleza-Jeric et al.
BMJ 2005;330:956-958.
FULL TEXT  

Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors
Chan and Altman
BMJ 2005;330:753.
ABSTRACT | FULL TEXT  

Extended-Interval Aminoglycosides in Children: More Guidance Is Needed
Best et al.
Pediatrics 2005;115:827-828.
FULL TEXT  

Trial protocols at the BMJ
Jones and Abbasi
BMJ 2004;329:1360-1360.
FULL TEXT  

The Consolidated Standards of Reporting Trials (CONSORT): Better Presentation of Surgical Trials in the Journal of Hand Surgery
SAUERLAND and DAVIS
J Hand Surg Eur Vol 2004;29:621-624.
ABSTRACT | FULL TEXT  

Better Reporting of Harms in Randomized Trials: An Extension of the CONSORT Statement
Ioannidis et al.
ANN INTERN MED 2004;141:781-788.
ABSTRACT | FULL TEXT  

Publication bias and journals as policemen
DeMaria
J Am Coll Cardiol 2004;44:1707-1708.
FULL TEXT  

Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan et al.
CMAJ 2004;171:735-740.
ABSTRACT | FULL TEXT  

Registering CIHR-funded randomized controlled trials: a global public good
Moher and Bernstein
CMAJ 2004;171:750-751.
FULL TEXT  

Trial Registration: A Great Idea Switches From Ignored to Irresistible
Rennie
JAMA 2004;292:1359-1362.
FULL TEXT