This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (294)  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Topic Collections  • Journalology/ Peer Review/ Authorship  • Statistics and Research Methods  • Randomized Controlled Trial  • Prognosis/ Outcomes  • Alert me on articles by topic  Social Bookmarking   What's this?

Comparison of Protocols to Published Articles

An-Wen Chan, MD, DPhil; Asbjørn Hróbjartsson, MD, PhD; Mette T. Haahr, BSc; Peter C. Gøtzsche, MD, DrMedSci; Douglas G. Altman, DSc

JAMA. 2004;291:2457-2465.

Context  Selective reporting of outcomes within published studies based on the nature or direction of their results has been widely suspected, but direct evidence of such bias is currently limited to case reports.

Objective  To study empirically the extent and nature of outcome reporting bias in a cohort of randomized trials.

Design  Cohort study using protocols and published reports of randomized trials approved by the Scientific-Ethical Committees for Copenhagen and Frederiksberg, Denmark, in 1994-1995. The number and characteristics of reported and unreported trial outcomes were recorded from protocols, journal articles, and a survey of trialists. An outcome was considered incompletely reported if insufficient data were presented in the published articles for meta-analysis. Odds ratios relating the completeness of outcome reporting to statistical significance were calculated for each trial and then pooled to provide an overall estimate of bias. Protocols and published articles were also compared to identify discrepancies in primary outcomes.

Main Outcome Measures  Completeness of reporting of efficacy and harm outcomes and of statistically significant vs nonsignificant outcomes; consistency between primary outcomes defined in the most recent protocols and those defined in published articles.

Results  One hundred two trials with 122 published journal articles and 3736 outcomes were identified. Overall, 50% of efficacy and 65% of harm outcomes per trial were incompletely reported. Statistically significant outcomes had a higher odds of being fully reported compared with nonsignificant outcomes for both efficacy (pooled odds ratio, 2.4; 95% confidence interval [CI], 1.4-4.0) and harm (pooled odds ratio, 4.7; 95% CI, 1.8-12.0) data. In comparing published articles with protocols, 62% of trials had at least 1 primary outcome that was changed, introduced, or omitted. Eighty-six percent of survey responders (42/49) denied the existence of unreported outcomes despite clear evidence to the contrary.

Conclusions  The reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols. Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention. To ensure transparency, planned trials should be registered and protocols should be made publicly available prior to trial completion.

Author Affiliations: Centre for Statistics in Medicine, Institute of Health Sciences, Oxford, England (Drs Chan and Altman), Nordic Cochrane Centre, Copenhagen, Denmark (Drs Hróbjartsson and Gøtzsche and Ms Haahr), University Health Network, University of Toronto, Toronto, Ontario (Dr Chan).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Clinical Trials Design Lessons From the CATIE Study
Kraemer et al.
Am. J. Psychiatry 2009;166:1222-1228.
ABSTRACT | FULL TEXT  

Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling et al.
BMJ 2009;339:b4012-b4012.
ABSTRACT | FULL TEXT  

Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses
Ioannidis
CMAJ 2009;181:488-493.
FULL TEXT  

The tyranny of power: is there a better way to calculate sample size?
Bland
BMJ 2009;339:b3985-b3985.
FULL TEXT  

Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials
Mathieu et al.
JAMA 2009;302:977-984.
ABSTRACT | FULL TEXT  

Better regulation of industry-sponsored clinical trials is long overdue.
Wynia and Boren
J Law Med Ethics 2009;37:410-419.
 

Reprint--Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher et al.
ptjournal 2009;89:873-880.
ABSTRACT | FULL TEXT  

Conflicts of Interest, Authorship, and Disclosures in Industry-Related Scientific Publications: The Tort Bar and Editorial Oversight of Medical Journals
Hirsch
Mayo Clin Proc. 2009;84:811-821.
FULL TEXT  

Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher et al.
ANN INTERN MED 2009;151:264-269.
FULL TEXT  

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration
Liberati et al.
ANN INTERN MED 2009;151:W-65-W-94.
ABSTRACT | FULL TEXT  

Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study
Tendal et al.
BMJ 2009;339:b3128-b3128.
ABSTRACT | FULL TEXT  

Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications
Moreno et al.
BMJ 2009;339:b2981-b2981.
ABSTRACT | FULL TEXT  

Interpretation of Randomized Controlled Trials of Fracture Prevention
Nguyen
IBMS BoneKEy 2009;6:279-294.
ABSTRACT | FULL TEXT  

Invited review: Use of meta-analysis in animal health and reproduction: Methods and applications
Lean et al.
J DAIRY SCI 2009;92:3545-3565.
ABSTRACT | FULL TEXT  

Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement
Moher et al.
BMJ 2009;339:b2535-b2535.
FULL TEXT  

The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration
Liberati et al.
BMJ 2009;339:b2700-b2700.
ABSTRACT | FULL TEXT  

Searching for Genes That Matter in Acute Kidney Injury: A Systematic Review
Lu et al.
CJASN 2009;4:1020-1031.
ABSTRACT | FULL TEXT  

Reporting of sample size calculation in randomised controlled trials: review
Charles et al.
BMJ 2009;338:b1732-b1732.
ABSTRACT | FULL TEXT  

Patients and the public deserve big changes in evaluation of drugs
Garattini and Chalmers
BMJ 2009;338:b1025-b1025.
FULL TEXT  

Industry attack on academics
Gornall
BMJ 2009;338:b736-b736.
FULL TEXT  

Alitretinoin as a potential advance in the management of severe chronic hand eczema.
Ingram et al.
Arch Dermatol 2009;145:314-315.
FULL TEXT  

Low intensity pulsed ultrasonography for fractures: systematic review of randomised controlled trials
Busse et al.
BMJ 2009;338:b351-b351.
ABSTRACT | FULL TEXT  

STrengthening the REporting of Genetic Association Studies (STREGA): An Extension of the STROBE Statement
Little et al.
ANN INTERN MED 2009;150:206-215.
ABSTRACT | FULL TEXT  

Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?
Thorlund et al.
Int J Epidemiol 2009;38:276-286.
ABSTRACT | FULL TEXT  

Apparently conclusive meta-analyses may be inconclusive--Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses
Brok et al.
Int J Epidemiol 2009;38:287-298.
ABSTRACT | FULL TEXT  

Commentary: Which meta-analyses are conclusive?
Nuesch and Juni
Int J Epidemiol 2009;38:298-303.
FULL TEXT  

Acupuncture treatment for pain: systematic review of randomised clinical trials with acupuncture, placebo acupuncture, and no acupuncture groups
Madsen et al.
BMJ 2009;338:a3115-a3115.
ABSTRACT | FULL TEXT  

Reporting and interpretation of SF-36 outcomes in randomised trials: systematic review
Contopoulos-Ioannidis et al.
BMJ 2009;338:a3006-a3006.
ABSTRACT | FULL TEXT  

Study Quality Assessment in Systematic Reviews of Research on Intervention Effects
Wells and Littell
Research on Social Work Practice 2009;19:52-62.
ABSTRACT  

Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols
Chan et al.
BMJ 2008;337:a2299-a2299.
ABSTRACT | FULL TEXT  

Selective Reporting of Adjusted Estimates in Observational Epidemiology Studies: Reasons and Implications for Meta-analyses
Peters and Mengersen
Eval Health Prof 2008;31:370-389.
ABSTRACT  

Evaluation of the Potential Excess of Statistically Significant Findings in Published Genetic Association Studies: Application to Alzheimer's Disease
Kavvoura et al.
Am J Epidemiol 2008;168:855-865.
ABSTRACT | FULL TEXT  

Commercialism, Loss of Professionalism, and the Effect on Journals
Liesegang
Arch Ophthalmol 2008;126:1292-1295.
FULL TEXT  

Reporting Bias in Diagnostic and Prognostic Studies: Time for Action
Rifai et al.
Clin. Chem. 2008;54:1101-1103.
FULL TEXT  

What is missing from descriptions of treatment in trials and reviews?
Glasziou et al.
BMJ 2008;336:1472-1474.
FULL TEXT  

Patient-Important Outcomes in Registered Diabetes Trials
Gandhi et al.
JAMA 2008;299:2543-2549.
ABSTRACT | FULL TEXT  

Research Reporting and Evidence of Effectiveness: Why "No Difference" Matters
Hackbarth
Am J Crit Care 2008;17:218-220.
FULL TEXT  

How Much of the Data Published in Observational Studies of the Association between Diet and Prostate or Bladder Cancer Is Usable for Meta-Analysis?
Bekkering et al.
Am J Epidemiol 2008;167:1017-1026.
ABSTRACT | FULL TEXT  

PROactive: time for a critical appraisal
Betteridge et al.
Eur Heart J 2008;29:969-983.
ABSTRACT | FULL TEXT  

Publishing study protocols
Johnston
Inj. Prev. 2008;14:73-73.
FULL TEXT  

Exploring the Geometry of Treatment Networks
Salanti et al.
ANN INTERN MED 2008;148:544-553.
ABSTRACT | FULL TEXT  

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
Wood et al.
BMJ 2008;336:601-605.
ABSTRACT | FULL TEXT  

A Case for Clarity, Consistency, and Helpfulness: State-of-the-Art Clinical Practice Guidelines in Endocrinology Using the Grading of Recommendations, Assessment, Development, and Evaluation System
Swiglo et al.
J. Clin. Endocrinol. Metab. 2008;93:666-673.
ABSTRACT | FULL TEXT  

Moving from opacity to transparency in pharmaceutical policy
Dhalla and Laupacis
CMAJ 2008;178:428-431.
FULL TEXT  

Improving access to research data in Europe
Mladovsky et al.
BMJ 2008;336:287-288.
FULL TEXT  

Adding aripiprazole improves major depressive disorder following incomplete response to antidepressants alone
Cipriani and Barbui
Evid. Based Ment. Health 2008;11:15-15.
FULL TEXT  

Assessment of cumulative evidence on genetic associations: interim guidelines
Ioannidis et al.
Int J Epidemiol 2008;37:120-132.
ABSTRACT | FULL TEXT  

Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy
Turner et al.
NEJM 2008;358:252-260.
ABSTRACT | FULL TEXT  

RE: "INVITED COMMENTARY: HORMONE THERAPY AND RISK OF CORONARY HEART DISEASE WHY RENEW THE FOCUS ON THE EARLY YEARS OF MENOPAUSE?"
Doren and Greiser
Am J Epidemiol 2007;166:1479-1480.
FULL TEXT  

State of the Evidence on Acute Asthma Management in Children: A Critical Appraisal of Systematic Reviews
Boluyt et al.
Pediatrics 2007;120:1334-1343.
ABSTRACT | FULL TEXT  

Increased Mortality, Postoperative Morbidity, and Cost After Red Blood Cell Transfusion in Patients Having Cardiac Surgery
Murphy et al.
Circulation 2007;116:2544-2552.
ABSTRACT | FULL TEXT  

Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration
Vandenbroucke et al.
ANN INTERN MED 2007;147:W-163-W-194.
ABSTRACT | FULL TEXT  

Claims of Sex Differences: An Empirical Assessment in Genetic Associations
Patsopoulos et al.
JAMA 2007;298:880-893.
ABSTRACT | FULL TEXT  

Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal et al.
Int J Epidemiol 2007;36:847-857.
ABSTRACT | FULL TEXT  

An exploratory test for an excess of significant findings
Ioannidis and Trikalinos
Clin Trials 2007;4:245-253.
ABSTRACT  

Authors' response to V Johnson and Y Yuan
Ioannidis and Trikalinos
Clin Trials 2007;4:256-257.
 

Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding
Hrobjartsson et al.
Int J Epidemiol 2007;36:654-663.
ABSTRACT | FULL TEXT  

Issues in the Registration of Clinical Trials
Zarin et al.
JAMA 2007;297:2112-2120.
ABSTRACT | FULL TEXT  

Ethical Considerations in the Interpretation and Communication of Clinical Trial Results
Coultas
Proc Am Thorac Soc 2007;4:194-199.
ABSTRACT | FULL TEXT  

Genetic Effects versus Bias for Candidate Polymorphisms in Myocardial Infarction: Case Study and Overview of Large-Scale Evidence
Ntzani et al.
Am J Epidemiol 2007;165:973-984.
ABSTRACT | FULL TEXT  

Reporting of radiographic methods in randomised controlled trials assessing structural outcomes in rheumatoid arthritis
Baron et al.
Ann Rheum Dis 2007;66:651-657.
ABSTRACT | FULL TEXT  

Effects of platelet glycoprotein IIb/IIIa receptor blockers in non-ST segment elevation acute coronary syndromes: benefit and harm in different age subgroups
Hernandez et al.
Heart 2007;93:450-455.
ABSTRACT | FULL TEXT  

Oral presentation bias: a retrospective cohort study
Decullier and Chapuis
J. Epidemiol. Community Health 2007;61:190-193.
ABSTRACT | FULL TEXT  

Association Between Unreported Outcomes and Effect Size Estimates in Cochrane Meta-analyses
Furukawa et al.
JAMA 2007;297:468-470.
FULL TEXT  

Meta-analysis: Intravenous Immunoglobulin in Critically Ill Adult Patients with Sepsis
Turgeon et al.
ANN INTERN MED 2007;146:193-203.
ABSTRACT | FULL TEXT  

The Use of the World Wide Web by Medical Journals in 2003 and 2005: An Observational Study
Schriger et al.
Pediatrics 2007;119:e53-e60.
ABSTRACT | FULL TEXT  

'They would say that, wouldn't they?' A reader's guide to author and sponsor biases in clinical research
Bastian
JRSM 2006;99:611-614.
FULL TEXT  

Clinical trial registration at Tobacco Control
Bero
Tobacco Control 2006;15:417-418.
FULL TEXT  

Latency of the mfVEP to diagnose glaucoma?
Bach
Br J Ophthalmol 2006;90:1076-1077.
FULL TEXT  

Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors
Haahr and Hrobjartsson
Clin Trials 2006;3:360-365.
ABSTRACT  

Believability of relative risks and odds ratios in abstracts: cross sectional study
Gotzsche
BMJ 2006;333:231-234.
ABSTRACT | FULL TEXT  

Commentary: Grading the credibility of molecular evidence for complex diseases
Ioannidis
Int J Epidemiol 2006;35:572-578.
FULL TEXT  

Reported outcomes in major cardiovascular clinical trials funded by for-profit and not-for-profit organizations: 2000-2005.
Ridker and Torres
JAMA 2006;295:2270-2274.
ABSTRACT | FULL TEXT  

Waiving confidentiality for the greater good.
Chan et al.
BMJ 2006;332:1086-1089.
FULL TEXT  

Constraints on Publication Rights in Industry-Initiated Clinical Trials
Gotzsche et al.
JAMA 2006;295:1645-1646.
FULL TEXT  

Bias in Clinical Intervention Research
Gluud
Am J Epidemiol 2006;163:493-501.
ABSTRACT | FULL TEXT  

Quality of Reporting of Noninferiority and Equivalence Randomized Trials
Le Henanff et al.
JAMA 2006;295:1147-1151.
ABSTRACT | FULL TEXT  

Lessons From and Cautions About Noninferiority and Equivalence Randomized Trials
Gotzsche
JAMA 2006;295:1172-1174.
FULL TEXT  

On bias and transparency in the development of influential recommendations
Laupacis
CMAJ 2006;174:335-336.
FULL TEXT  

A Call for Fairness in Formulary Decisions
Levinson and Laupacis
Arch Intern Med 2006;166:16-18.
FULL TEXT  

Registries and Registration of Clinical Trials
Haug et al.
NEJM 2005;353:2811-2812.
FULL TEXT  

Outcome selection bias in meta-analysis
Williamson et al.
Stat Methods Med Res 2005;14:515-524.
ABSTRACT