 |
|
Positive-Outcome Bias and Other Limitations in the Outcome of Research Abstracts Submitted to a Scientific Meeting
Michael L. Callaham, MD;
Robert L. Wears, MD;
Ellen J. Weber, MD;
Christopher Barton, MD;
Gary Young, MD
JAMA. 1998;280:254-257.
ABSTRACT
| |
Context.— Studies with positive results are more likely to be published in biomedical journals than are studies with negative results. However, many studies submitted for consideration at scientific meetings are never published in full; bias in this setting is poorly studied.
Objective.— To identify features associated with the fate of research abstracts submitted to a scientific meeting.
Design and Setting.— Prospective observational cohort, with 5-year follow-up of all research submitted for consideration to the major annual 1991 US research meeting in the specialty of emergency medicine.
Participants.— All research abstracts submitted for consideration at the meeting for possible presentation.
Main Outcome Measures.— Characteristics associated with acceptance for presentation at the meeting and subsequent publication as a full manuscript.
Results.— A total of 492 research abstracts were submitted from programs in emergency medicine and other specialies affiliated with 103 US medical schools. A total of 179 (36%) were accepted for presentation and 214 (43%) were published in 44 journals. Of the 179 abstracts accepted for presentation, 111 studies were published. Scientific quality of abstracts or prestige of the journal in which the study was eventually published did not predict either of these outcomes. The best predictors (by logistic regression) of meeting acceptance were a subjective "originality" factor (odds ratio [OR], 2.07; 95% confidence interval [CI], 1.13-3.89) and positive results (OR, 1.99; 95% CI, 1.07-3.84), and, for publication, meeting acceptance (OR, 2.49; 95% CI, 1.49-4.35) and large sample size (OR, 2.26; 95% CI, 1.23-4.31). Forty-nine percent (241) of abstracts did not report on blinding, and 24% (118) did not report on randomization. Acceptance and publication were both more likely for positive outcomes (P=.03). Funnel plots showed the classic distribution of positive-outcome ("publication") bias at each of the submission, acceptance, and publication phases. Meeting acceptance predicted publication with a sensitivity of only 51%, specificity of 71%, positive predictive value of 57%, and negative predictive value of 66%.
Conclusions.— Positive-outcome bias was evident when studies were submitted for consideration and was amplified in the selection of abstracts for both presentation and publication, neither of which was strongly related to study design or quality.
INTRODUCTION
POSITIVE-OUTCOME (also known as "publication") bias refers to the fact that research with positive outcomes is much more likely to be published than that with negative outcomes.1-4 Presentation of results in abstracts at scientific meetings is the first and often only publication for most biomedical research studies.5 However, the abstract selection process for meetings rarely has been studied. We, therefore, examined all research submitted for presentation at a national meeting to determine if positive-outcome bias was present in this process and what characteristics determined successful subsequent publication in a peer review journal.
METHODS
The Society for Academic Emergency Medicine (SAEM) meeting is comparable to the meetings of 31 other societies of the Council of Academic Societies.6 Abstracts with mandatory structured formats were submitted and, independent of our study (and similar to other specialty meetings), each submission was evaluated by 5 to 7 blinded members of the SAEM screening committee (selected for their relevant expertise) and ranked on a 5-point Likert scale. Selection was based on an average score, and no journal had right of first refusal.
Four years after the 21st annual meeting of SAEM in 1991, all SAEM data were obtained by the authors and each submitted study was categorized according to design by a blinded Delphi panel. Since no established system exists for abstract classification, we modified a previously published approach.7 In addition, the review panel ranked each study for scientific quality and "originality" ("newsworthiness") on a Likert scale like the one previously validated.8 Institutional review board approval was obtained, and a detailed description of the methods used is available from the authors.
All authors' names were searched in MEDLINE in late 1995 to determine if the study had been published in any listed journal.9 For papers not found, the search was repeated in early 1996; if still not found, a questionnaire was sent to the authors and EMBASE and the Cochrane Collaboration databases were also searched.9 Journal impact factor was derived from the Science Citation Index for the year of publication.10-13 The authors' institutions were ranked according to a system14-15 based on total dollars of National Institutes of Health (NIH) grant support.
There is no standardized definition of positive results.16 We used one of the more common definitions—that results were positive if the studied variable produced positive (beneficial) results.1, 3, 17-18 Some authors have defined positive results as those reporting statistically significant results (regardless of direction),4-5,16, 19-21 so we also used this definition.
We performed the major logistic regression analysis on those studies in which the subjects were either humans or animals, and the design was a prospective interventional trial, a prospective observational study, or a retrospective observational study. We also separately examined prospective studies with controls, excluding retrospective studies. We used a general iterative model-building strategy, as suggested by Hosmer and Lemeshow22 and Harrell,23 assessed for goodness of fit and subjected to bootstrap validation using S-Plus version 4.0, release 3 (Mathsoft Inc, Seattle, Wash) and Harrell extensions (http://www.lib.stat.cmu.edu). Correlation coefficients were calculated using JMP software 3.2.2 for Macintosh (SAS Institute, Inc, Cary, NC). We calculated the effect size of all interventions for the subgroup of prospective studies with controls in the usual fashion, except that SDs were not available.
RESULTS
Five hundred research abstracts were submitted to the SAEM selection committee from a total of 144 institutions, 103 of which had formal US medical school affiliations. Eight duplicates were deleted, leaving 492 abstracts as the basis of our study. The submitting schools averaged an NIH funding rank of 55 (of all US medical schools) (95% confidence interval [CI], 49-62).
Thirty percent of studies did not state a hypothesis, and 49% did not report on blinding, 24% on randomization, and 74% on exclusion criteria. Seventy-six percent of the studies were conducted on humans, 10% on animals, and the remainder on other models. Twenty-nine percent of the studies were retrospective, 27% prospective observational with control groups, and 26% prospective interventional trials. Sixty-six percent of all submitted interventional trials had positive outcome by our initial definition, 83% by the positive P value definition, and 80% by effect size. Respective figures for all submitted observational prospective studies were 70%, 92%, and 80%.
Three hundred eighty studies met the criteria for logistic regression (see "Methods"). Most measures of scientific merit did not predict the decision to accept an abstract for presentation. Instead, this decision was most strongly related to positive results and the reviewers' subjective originality score, while controlling for institutional funding, study design, randomization, blinding, controls, exclusion criteria, and sample size (Table 1). Results were similar for the subgroup of 166 prospective studies with control groups except that sample size greater than 50 was also predictive (odds ratio [OR], 2.4; 95% CI, 1.0-6.5) and similarly controlled. Full regression results are available from the authors.
|
|
|
|
Table 1.—Characteristics Predicting Acceptance for Presentation in Main Group of 380 Submitted Studies*
|
|
|
One hundred seventy-nine (36%) of the submitted abstracts were accepted for presentation at the meeting (11 reports on teaching methods were excluded). The SAEM committee scores determining acceptance correlated best with our subjective quality scale (R=0.57) and originality factor (R=0.49). Two hundred fourteen (43%) of the 492 studies submitted were published, an average of 18 months after presentation, in 44 journals with impact factors ranging from 0.23 to 24.5. A follow-up questionnaire to authors of unpublished papers was returned by 226 authors and identified 21 publications not found in MEDLINE.9 One hundred four (49%) of the 214 studies ultimately published were rejected for presentation at the meeting. The mean impact factor of the publishing journal did not differ for those papers rejected for the meeting vs those accepted (1.48 vs 1.19; P =.47), nor did time to publication.
One hundred forty-seven studies (70%) were published in emergency medicine specialty journals. The remaining studies were published in 39 other journals, including American Journal of Public Health, Annals of Internal Medicine, JAMA, The New England Journal of Medicine, Pediatrics, and Stroke. Of all the studies published after the meeting, 38% had been published 1 year later, 68% in 2 years, 88% in 3 years, and 95% in 4 years.
Publication of a full manuscript in the main group of 380 studies was related most strongly to abstract acceptance and sample size, controlling for the previous variables by logistic regression (Table 2). Results in the 166 prospective studies with control groups were identical. Results of these analyses did not differ using either of the 2 definitions of positive outcome ("Methods").
|
|
|
|
Table 2.—Characteristics Predicting Peer-Reviewed Publication of a Manuscript in Main Group of 380 Submitted Studies*
|
|
|
Data allowing calculation of effect size was reported in only 122 (66%) of 186 prospective studies. Positive-outcome bias was evident for all studies submitted before any screening (Figure 1). Submissions with less positive effect size were then disproportionately rejected for presentation. At the level of publication of full manuscripts, the same bias again appeared. A funnel plot (Figure 1) shows the absence of expected negative effect sizes at low sample size, which is the hallmark of positive-outcome bias.
|
|
|
|
Funnel plot of all 122 prospective studies submitted to the meeting that reported effect sizes (see text). Studies subsequently accepted for presentation at the meeting are plotted separately from those rejected. Inset graph similarly shows results at the journal publication level. The absence of the left (negative) base of the inverted funnel suggests the presence of selection bias in favor of positive outcomes.
|
|
|
Numerical testing confirms the funnel plots. The mean effect size of all submitted papers was 0.71 (95% CI, 0.40-1.01). The mean effect size of papers accepted for the meeting was 0.92 vs 0.45 for those rejected. The mean effect size of papers eventually published was 0.96 vs 0.45 for those never published (P=.03, Kruskal-Wallis analysis of variance). Effect size contributed much more to acceptance or publication than study sample size.
LIMITATIONS
Our study was limited to 1 specialty, but 103 medical schools contributed and the publication rate was comparable with 31 other academic society meetings.6 Research from this meeting was published in 44 journals, 39 of them outside this specialty. The emergency medicine literature, the abstracts we studied, and the general medicine literature are identical in the proportion of studies with positive outcomes.1 Effect size data were available for only a minority of studies, and most studies were not randomized controlled trials. These subgroups might not be representative of all 492 abstracts, but the results between groups were similar and consistent.
COMMENT
Presentation of scientific studies at meetings is an important part of the dissemination of knowledge, but half of these studies appear only as abstracts and never undergo any other peer review.5 Whether the abbreviated peer review used to select abstracts for meetings actually identifies scientific merit is unknown, yet abstracts are cited as often as fully published papers.24
We reviewed all submitted research, not just studies accepted for presentation, assessing those characteristics previously suggested to predict publication.2-5,14-15 Our results show that acceptance of an abstract for presentation at the meeting was not strongly related to study design, methods, sample size, or even a subjective quality score. Instead, a subjective "originality" factor and presence of positive results best predicted acceptance (ORs, 2.07 and 1.99, respectively), regardless of study design.
Publication as a full manuscript was best predicted by whether the abstract had been accepted at the meeting (OR, 2.49) and large sample size (OR, 2.26), again independent of study design or scientific quality. Positive studies were preferentially accepted during both the acceptance and publication decisions (P=.03), which is illustrated in the funnel plots (Figure 1).
Positive-outcome bias has been documented previously in publication of full journal articles, but not in detail at the meeting acceptance level.3-4,18 Four studies examined, in limited ways, the publication of studies after acceptance for presentation at meetings.5, 25-27 Abstracts submitted on the single subject of gestational exposure to cocaine demonstrated positive-outcome bias in acceptance.28 The impact of sample size and positive outcome on acceptance and publication was reported for cancer abstracts.24
Our study examined all submitted research from a broad cross-section of institutions, with subsequent publication in a broad variety of journals. Positive-effect (or publication) bias was already present when studies were first submitted for consideration (Figure 1). Presumably this was due to authors who did not complete or submit smaller studies with negative effects, perhaps after experiencing a tradition of publication bias by meeting selection committees and scientific journals.
The selection process for presentation at the meeting further increased this positive-outcome bias (Figure 1). Logistic regression showed that an intangible "originality" ("newsworthiness") factor and positive outcome were more strongly associated with acceptance than traditional measures of scientific quality, such as study design, randomization, sample size, and blinding (Table 1 and Table 2).
Positive-outcome bias appeared again in the selection process for publication in a journal. Full publication was best predicted by acceptance at the meeting and study size, rather than study methods or quality. Our results confirm a smaller study of the cancer literature, which did not control for scientific quality.24
A number of potential solutions, such as trials registries, have been proposed to remedy positive-outcome bias.21, 29-30 We offer one more solution: that all studies submitted to scientific meetings be published as abstracts, indicating whether or not they were chosen to be presented. This might encourage researchers to submit studies with negative findings, and readers and researchers could more easily identify the entire spectrum of research. Journals might adopt a similar practice, publishing the abstracts of all submitted manuscripts.
Despite the mandatory structured format, 49% of SAEM abstracts failed to report adequately about blinding, 74% about exclusion criteria, 24% about randomization, and 14% about sample size. Perhaps because these deficiencies made the merit of the research difficult to evaluate, acceptance for presentation at the meeting predicted publication as a full manuscript with a sensitivity of only 51%, a specificity of 71%, a positive predictive value of 57%, and a negative predictive value of 66%.
AUTHOR INFORMATION
Presented at the Third International Congress on Peer Review in Biomedical Publication, Prague, Czech Republic, September 19, 1997.
We wish to thank Mary Ann Schropp, executive director of the SAEM, for her generous help with data, and John Gallagher, MD, for sharing the institutional research funding data.
Reprints: Michael L. Callaham, MD, Box 0208, University of California, San Francisco, CA 94143-0208 (e-mail: mlc{at}itsa.ucsf.edu).
From the Division of Emergency Medicine (Drs Weber, Callaham, and Young), University of California, San Francisco; the Division of Emergency Medicine, University of Florida Health Center, Jacksonville (Dr Wears); the Department of Emergency Medicine, University of North Carolina, Chapel Hill (Dr Barton); and the Department of Emergency Medicine, Highland Hospital, Alameda County Medical Center, Oakland, Calif (Dr Young). Dr Young is now with Sacred Heart Medical Center, Eugene, Ore.
REFERENCES
1. Moscati R, Jehle D, Ellis D, Fiorello A, Landi M. Positive-outcome bias: comparison of emergency medicine and general medicine literatures. Acad Emerg Med. 1994;1:267-271.
PUBMED
2. Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr. Publication bias and clinical trials. Control Clin Trials. 1987;8:343-353.
FULL TEXT
|
ISI
| PUBMED
3. Dickersin K, Min Y-I, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA. 1992;267:374-378.
ABSTRACT
4. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337:867-872.
FULL TEXT
|
ISI
| PUBMED
5. Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts: a meta-analysis. JAMA. 1994;272:158-162.
ABSTRACT
6. Wuerz R, Holliman J. Attendance and research abstract activity at the 1993 annual meetings of the academic medical societies. Acad Emerg Med. 1994;1:A59.
7. Chalmers T, Smith H, Blackburn B, et al. A method for assessing the quality of a randomized controlled trial. Control Clin Trials. 1981;2:31-49.
FULL TEXT
|
ISI
| PUBMED
8. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271-1278.
FULL TEXT
|
ISI
| PUBMED
9. Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA. 1998;280:257-259.
FREE FULL TEXT
10. Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Pa: Institute for Science Information Inc; 1991.
11. Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Pa: Institute for Science Information Inc; 1992.
12. Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Pa: Institute for Science Information Inc; 1993.
13. Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Pa: Institute for Science Information Inc; 1994.
14. Gallagher E, Goldfrank L, Anderson G, et al. Current status of academic emergency medicine within academic medicine in the United States. Acad Emerg Med. 1994;1:41-46.
PUBMED
15. Garfunkel JM, Ulshen MH, Hamrick HJ, Lawson EE. Effect of institutional prestige on reviewers' recommendations and editorial decisions. JAMA. 1994;272:137-138.
ABSTRACT
16. Olson CM. Publication bias. Acad Emerg Med. 1994;1:207-209.
PUBMED
17. Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986;4:1529-1541.
FREE FULL TEXT
18. Dickersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clin Trials [serial online]. 1993;doc 50.
19. Rennie D, Flanagin A. Publication bias: the triumph of hope over experience. JAMA. 1992;267:411-412.
FULL TEXT
|
ISI
| PUBMED
20. Berlin JA. Will publication bias vanish in the age of online journals? Online J Curr Clin Trials [serial online]. 1992;doc 12.
21. Begg CB, Berlin JA. Publication bias and dissemination of clinical research. J Natl Cancer Inst. 1989;81:107-115.
FREE FULL TEXT
22. Hosmer D, Lemeshow S. Applied Logistic Regression. New York, NY: John Wiley & Sons Inc; 1989.
23. Harrell FE. Predicting Outcomes: Applied Survival Analysis and Logistic Regression. Charlottesville: Dept of Health Evaluation Sciences, School of Medicine, University of Virginia; 1996.
24. de Bellefeuille C, Morrison C, Tannock I. The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication. Ann Oncol. 1992;3:187-191.
FREE FULL TEXT
25. Goldman L, Loscalzo A. Fate of cardiology research originally published in abstract form. N Engl J Med. 1980;303:255-259.
ABSTRACT
26. Meranze J, Ellison N, Greenhow D. Publications reseulting from anesthesia meeting abstracts. Anesth Analg. 1982;61:445-448.
FREE FULL TEXT
27. McCormick M, Holmes J. Publication of research presented at the pediatric meetings. AJDC. 1985;139:122-126.
28. Koren G, Graham K, Shear H, Einarson T. Bias against the null hypothesis: the reproductive hazards of cocaine. Lancet. 1989;2:1440-1444.
FULL TEXT
|
ISI
| PUBMED
29. Newcombe RG. Towards a reduction in publication bias. BMJ. 1987;295:656-659.
ISI
| PUBMED
30. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:1385-1389.
ABSTRACT
31. Weintraub WH. Are published manuscripts representative of the surgical meeting abstracts? an objective appraisal. J Pediatr Surg. 1987;22:11-13.
ISI
| PUBMED
RELATED ARTICLE
July 15, 1998
JAMA. 1998;280(3):303-304.
EXTRACT
| FULL TEXT
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
|
Publication Bias in Orthopaedic Research: An Analysis of Scientific Factors Associated with Publication in The Journal of Bone and Joint Surgery (American Volume)
Okike et al.
JBJS 2008;90:595-601.
ABSTRACT
| FULL TEXT
Gating Properties of SCN5A Mutations and the Response to Mexiletine in Long-QT Syndrome Type 3 Patients
Ruan et al.
Circulation 2007;116:1137-1144.
ABSTRACT
| FULL TEXT
Eligibility Criteria of Randomized Controlled Trials--Reply
Fowler and Van Spall
JAMA 2007;298:39-40.
FULL TEXT
Commercially Funded and United States-Based Research Is More Likely to Be Published; Good-Quality Studies with Negative Outcomes Are Not
Lynch et al.
JBJS 2007;89:1010-1018.
ABSTRACT
| FULL TEXT
No Association Between the Common MTHFR 677C->T Polymorphism and Venous Thrombosis: Results From the MEGA Study
Bezemer et al.
Arch Intern Med 2007;167:497-501.
ABSTRACT
| FULL TEXT
Oral presentation bias: a retrospective cohort study
Decullier and Chapuis
J. Epidemiol. Community Health 2007;61:190-193.
ABSTRACT
| FULL TEXT
Frequency and Determinants of Nonpublication of Research in the Stroke Literature
Sanossian et al.
Stroke 2006;37:2588-2592.
ABSTRACT
| FULL TEXT
Critical appraisal and positive outcome bias in case reports published in brazilian dental journals.
Oliveira and Leles
J Dent Educ 2006;70:869-874.
ABSTRACT
| FULL TEXT
Clinical Trials Registries: A Reform that is Past Due
Gold and Studdert
J Law Med Ethics 2005;33:811-820.
Negative Results and Impact Factor: A Lesson From Neonatology
Littner et al.
Arch Pediatr Adolesc Med 2005;159:1036-1037.
ABSTRACT
| FULL TEXT
Outcome selection bias in meta-analysis
Williamson et al.
Stat Methods Med Res 2005;14:515-524.
ABSTRACT
Acute Endophthalmitis Following Cataract Surgery: A Systematic Review of the Literature
Taban et al.
Arch Ophthalmol 2005;123:613-620.
ABSTRACT
| FULL TEXT
Association Between Compensation Status and Outcome After Surgery: A Meta-analysis
Harris et al.
JAMA 2005;293:1644-1652.
ABSTRACT
| FULL TEXT
Factors Influencing the Publication of Randomized Controlled Trials in Child Health Research
Hartling et al.
Arch Pediatr Adolesc Med 2004;158:983-987.
ABSTRACT
| FULL TEXT
Subsequent Publication of Orally Presented Original Studies within 5 Years after 1995 RSNA Scientific Assembly
Arrive et al.
Radiology 2004;232:101-106.
ABSTRACT
| FULL TEXT
Industry-Sponsored Economic Studies in Critical and Intensive Care Versus Studies Sponsored by Nonprofit Organizations
Hartmann et al.
J Intensive Care Med 2003;18:265-268.
ABSTRACT
A Systematic Review of Diabetes Self-Care Interventions for Older, African American, or Latino Adults
Sarkisian et al.
The Diabetes Educator 2003;29:467-479.
ABSTRACT
Systematic reviews in epidemiology: why are we so far behind?
Dickersin
Int J Epidemiol 2002;31:6-12.
FULL TEXT
Commentary: Toward systematic reviews in epidemiology
Bracken
Int J Epidemiol 2001;30:954-957.
FULL TEXT
Research outcomes in British gastroenterology: an audit of the subsequent full publication of abstracts presented at the British Society of Gastroenterology
SANDERS et al.
Gut 2001;49:154a-155.
FULL TEXT
Prepublication Release of Medical Research
Fontanarosa and Flanagin
JAMA 2000;284:2927-2929.
FULL TEXT
Publication Bias Is a Scientific Problem with Adverse Ethical Outcomes: The Case for a Section for Null Results
Shields
Cancer Epidemiol. Biomarkers Prev. 2000;9:771-772.
FULL TEXT
Evaluation of Conflict of Interest in Economic Analyses of New Drugs Used in Oncology
Friedberg et al.
JAMA 1999;282:1453-1457.
ABSTRACT
| FULL TEXT
Evidence on peer review---scientific quality control or smokescreen?
Goldbeck-Wood
BMJ 1999;318:44-45.
FULL TEXT
Complementary Medicine and the Cochrane Collaboration
Ezzo et al.
JAMA 1998;280:1628-1630.
FULL TEXT
Peer Review in Prague
Rennie
JAMA 1998;280:214-215.
FULL TEXT
Unpublished Research From a Medical Specialty Meeting: Why Investigators Fail to Publish
Weber et al.
JAMA 1998;280:257-259.
ABSTRACT
| FULL TEXT
|
