This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (143)  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Topic Collections  • Journalology/ Peer Review/ Authorship  • Statistics and Research Methods  • Randomized Controlled Trial  • Alert me on articles by topic  Social Bookmarking   What's this?

John P. A. Ioannidis, MD

JAMA. 2005;294:218-228.

Context  Controversy and uncertainty ensue when the results of clinical research on the effectiveness of interventions are subsequently contradicted. Controversies are most prominent when high-impact research is involved.

Objectives  To understand how frequently highly cited studies are contradicted or find effects that are stronger than in other similar studies and to discern whether specific characteristics are associated with such refutation over time.

Design  All original clinical research studies published in 3 major general clinical journals or high-impact-factor specialty journals in 1990-2003 and cited more than 1000 times in the literature were examined.

Main Outcome Measure  The results of highly cited articles were compared against subsequent studies of comparable or larger sample size and similar or better controlled designs. The same analysis was also performed comparatively for matched studies that were not so highly cited.

Results  Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged. Five of 6 highly-cited nonrandomized studies had been contradicted or had found stronger effects vs 9 of 39 randomized controlled trials (P = .008). Among randomized trials, studies with contradicted or stronger effects were smaller (P = .009) than replicated or unchallenged studies although there was no statistically significant difference in their early or overall citation impact. Matched control studies did not have a significantly different share of refuted results than highly cited studies, but they included more studies with "negative" results.

Conclusions  Contradiction and initially stronger effects are not unusual in highly cited research of clinical interventions and their outcomes. The extent to which high citations may provoke contradictions and vice versa needs more study. Controversies are most common with highly cited nonrandomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones.

Author Affiliations: Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, and the Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Center, Boston, Mass.

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

'Yes' or 'No' to Routine Statins After Subarachnoid Hemorrhage to Prevent Delayed Cerebral Ischaemia, Vasospasm, and Death?: A Cautionary Tale of 2 Meta-Analyses
Sandercock
Stroke 2010;41:e1-e2.
FULL TEXT  

Reassessing the evidence for the link between dioxin and endometriosis: from molecular biology to clinical epidemiology
Guo et al.
Mol Hum Reprod 2009;15:609-624.
ABSTRACT | FULL TEXT  

Interpretation of Randomized Controlled Trials of Fracture Prevention
Nguyen
IBMS BoneKEy 2009;6:279-294.
ABSTRACT | FULL TEXT  

Clinical Practice Guidelines and Scientific Evidence
Concato and Horwitz
JAMA 2009;302:145-145.
FULL TEXT  

Adalimumab vs Methotrexate for the Treatment of Chronic Plaque Psoriasis
Batchelor et al.
Arch Dermatol 2009;145:704-706.
FULL TEXT  

When Wishful Thinking Leads to a Misty-Eyed Appraisal: The Story of the Adjuvant Colon Cancer Trials With Edrecolomab
Schmoll and Arnold
JCO 2009;27:1926-1929.
FULL TEXT  

Acknowledge and fix the multiple testing problem
Young
Int J Epidemiol 2009;0:dyp188v1-dyp188.
FULL TEXT  

Cereal-induced gender selection? Most likely a multiple testing false positive
Young et al.
Proc R Soc B 2009;276:1211-1212.
FULL TEXT  

Author's Response: The triumph of the null hypothesis: epidemiology in an age of change
Dunn
Int J Epidemiol 2009;38:607-607.
FULL TEXT  

Is Personality a Key Predictor of Missing Study Data? An Analysis From a Randomized Controlled Trial
Jerant et al.
Ann Fam Med 2009;7:148-156.
ABSTRACT | FULL TEXT  

How to Use an Article About Genetic Association: B: Are the Results of the Study Valid?
Attia et al.
JAMA 2009;301:191-197.
ABSTRACT | FULL TEXT  

Beyond the Randomized Clinical Trial: The Role of Effectiveness Studies in Evaluating Cardiovascular Therapies
Nallamothu et al.
Circulation 2008;118:1294-1303.
FULL TEXT  

False-Positive Results in Cancer Epidemiology: A Plea for Epistemological Modesty
Boffetta et al.
JNCI J Natl Cancer Inst 2008;100:988-995.
ABSTRACT | FULL TEXT  

Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better
Cumming
Perspectives on Psychological Science 2008;3:286-300.
ABSTRACT | FULL TEXT  

Should we use large scale healthcare interventions without clear evidence that benefits outweigh costs and harms? Yes
Crump
BMJ 2008;336:1276-1276.
FULL TEXT  

In Reply: Can GnRH Analogues Preserve Gonadal Function Without Preserving Fertility?
Sonmezer et al.
The Oncologist 2008;13:615-617.
FULL TEXT  

Re: Low-Fat Dietary Pattern and Cancer Incidence in the Women's Health Initiative Dietary Modification Randomized Controlled Trial
Young
JNCI J Natl Cancer Inst 2008;100:284-284.
FULL TEXT  

Translation of Genetic Discoveries into Clinical Therapies
Hackam and Hackam
ANN INTERN MED 2008;148:246-247.
FULL TEXT  

Systematic Review and Meta-analysis: When One Study Is Just not Enough
Garg et al.
CJASN 2008;3:253-260.
FULL TEXT  

Effects of Angiotensin-Converting Enzyme Inhibition in Low-Risk Patients Early After Coronary Artery Bypass Surgery
Rouleau et al.
Circulation 2008;117:24-31.
ABSTRACT | FULL TEXT  

Persistence of Contradicted Claims in the Literature
Tatsioni et al.
JAMA 2007;298:2517-2526.
ABSTRACT | FULL TEXT  

Should Meta-Analyses of Interventions Include Observational Studies in Addition to Randomized Controlled Trials? A Critical Examination of Underlying Principles
Shrier et al.
Am J Epidemiol 2007;166:1203-1209.
ABSTRACT | FULL TEXT  

Cooling for Neonatal Hypoxic Ischemic Encephalopathy: Do We Have the Answer?
Kirpalani et al.
Pediatrics 2007;120:1126-1130.
FULL TEXT  

Toxicity of 2,3,7,8-Tetrachlorodibenzo-p-dioxin in the Developing Male Wistar(Han) Rat. I: No Decrease in Epididymal Sperm Count after a Single Acute Dose
Bell et al.
Toxicol Sci 2007;99:214-223.
ABSTRACT | FULL TEXT  

How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania et al.
ANN INTERN MED 2007;147:224-233.
ABSTRACT | FULL TEXT  

Chronic Kidney Disease and the Public Health: Gaps in Evidence From Interventional Trials
Himmelfarb
JAMA 2007;297:2630-2633.
FULL TEXT  

Comments on `An exploratory test for an excess of significant findings' by JPA loannidis and TA Trikalinos
Johnson and Yuan
Clin Trials 2007;4:254-255.
 

An Ongoing Study of Anemia Correction in Chronic Kidney Disease
Pfeffer and the TREAT Executive Committee
NEJM 2007;356:959-961.
FULL TEXT  

Major Depression Symptoms in Primary Care and Psychiatric Care Settings: A Cross-Sectional Analysis
Gaynes et al.
Ann Fam Med 2007;5:126-134.
ABSTRACT | FULL TEXT  

A Rapid-Learning Health System
Etheredge
Health Aff (Millwood) 2007;26:w107-w118.
ABSTRACT | FULL TEXT  

Inferential, robust non-negative matrix factorization analysis of microarray data
Fogel et al.
Bioinformatics 2007;23:44-49.
ABSTRACT | FULL TEXT  

Current controversies in data monitoring for clinical trials
Pocock
Clin Trials 2006;3:513-521.
ABSTRACT  

Jottings...
Glasziou
Evid. Based Med. 2006;11:162-162.
FULL TEXT  

'They would say that, wouldn't they?' A reader's guide to author and sponsor biases in clinical research
Bastian
JRSM 2006;99:611-614.
FULL TEXT  

Attenti Alle Bufale ('Beware of Red Herrings'), or, How to Make Evidence-Based Medicine Work for You
Jefferson et al.
JRSM 2006;99:625-627.
FULL TEXT  

Pro: Aprotinin Has a Good Efficacy and Safety Profile Relative to Other Alternatives for Prevention of Bleeding in Cardiac Surgery
Body and Mazer
Anesth. Analg. 2006;103:1354-1359.
FULL TEXT  

Early goal-directed therapy in severe sepsis and septic shock revisited: concepts, controversies, and contemporary findings.
Otero et al.
Chest 2006;130:1579-1595.
ABSTRACT | FULL TEXT  

Translation of research evidence from animals to humans.
Hackam and Redelmeier
JAMA 2006;296:1731-1732.
FULL TEXT  

The sound and the fury.
Manthous
Am. J. Respir. Crit. Care Med. 2006;173:1414-1414.
FULL TEXT  

Commentary: Grading the credibility of molecular evidence for complex diseases
Ioannidis
Int J Epidemiol 2006;35:572-578.
FULL TEXT  

Origin and funding of the most frequently cited papers in medicine: database analysis
Patsopoulos et al.
BMJ 2006;332:1061-1064.
ABSTRACT | FULL TEXT  

Clinical Practice Guidelines in nephrology--for worse or for better
Uhlig et al.
Nephrol Dial Transplant 2006;21:1145-1153.
FULL TEXT  

Transition From Meeting Abstract to Full-length Journal Article for Randomized Controlled Trials
Toma et al.
JAMA 2006;295:1281-1287.
ABSTRACT | FULL TEXT  

Put my name on that paper: reflections on the ethics of authorship.
J. Thorac. Cardiovasc. Surg. 2006;131:517-519.
 

Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies
Papanikolaou et al.
CMAJ 2006;174:635-641.
ABSTRACT | FULL TEXT  

Contradictions in Highly Cited Clinical Research
West
JAMA 2005;294:2695-2695.
FULL TEXT  

Contradictions in Highly Cited Clinical Research
Heckerling
JAMA 2005;294:2695-2695.
FULL TEXT  

Contradictions in Highly Cited Clinical Research
Westfall and Young
JAMA 2005;294:2695-2696.
FULL TEXT  

When (Not) to Stop a Clinical Trial for Benefit
Pocock
JAMA 2005;294:2228-2230.
FULL TEXT  

Don't Believe Everything You Read
JWatch Emergency Med. 2005;2005:4-4.
FULL TEXT  

The Impact of Impacts
Journal Watch Dermatology 2005;2005:5-5.
FULL TEXT