 |
|
Accuracy of Data in Abstracts of Published Research Articles
Roy M. Pitkin, MD;
Mary Ann Branagan;
Leon F. Burmeister, PhD
JAMA. 1999;281:1110-1111.
ABSTRACT
| |
Context The section of a research article most likely to be read is the abstract, and therefore it is particularly important that the abstract reflect the article faithfully.
Objective To assess abstracts accompanying research articles published in 6 medical journals with respect to whether data in the abstract could be verified in the article itself.
Design Analysis of simple random samples of 44 articles and their accompanying abstracts published during 1 year (July 1, 1996-June 30, 1997) in each of 5 major general medical journals (Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine) and a consecutive sample of 44 articles published during 15 months (July 1, 1996-August 15, 1997) in the CMAJ.
Main Outcome Measure Abstracts were considered deficient if they contained data that were either inconsistent with corresponding data in the article's body (including tables and figures) or not found in the body at all.
Results The proportion of deficient abstracts varied widely (18%-68%) and to a statistically significant degree (P<.001) among the 6 journals studied.
Conclusions Data in the abstract that are inconsistent with or absent from the article's body are common, even in large-circulation general medical journals.
INTRODUCTION
The abstract accompanying a research article, because it is often the only part of the article that will be read, should reflect fully and accurately the work reported. We observed in 1 medical specialty journal that a quarter or more of manuscripts returned after revision contained data in the abstract that could not be verified in the body of the paper.1 If this problem were to persist in published articles, then a potential for misinterpretation would exist. In the present study, we surveyed research articles and their accompanying abstracts published recently in 6 medical journals to verify data in the abstract by relating them to corresponding data in the body of the report.
METHODS
Articles studied included simple random samples of reports of original research (including meta-analyses but not other types of reviews) appearing in 5 medical journals between July 1, 1996, and June 30, 1997 (Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine); all articles appearing in a sixth journal CMAJ (Canadian Medical Association Journal), between July 1, 1996, and August 15, 1997, were also studied. Additional inclusion criteria were (1) the article was accompanied by an abstract and (2) the article occupied at least 2 full journal pages.
To estimate the sample sizes, we used some preliminary observations1 that 25% to 50% of articles published in 2 of the journals studied contained 1 or more deficiencies in abstracts. We assumed this rate would range from 10% to 40% across the 6 journals studied and that  was .05 and power was 0.8, yielding a projected sample size of 44 from each journal. From each of the 5 journals that published more than 44 research articles in the 2 volumes studied (July 1, 1996-June 30, 1997), we selected a computer-generated simple random sample of 44. From the CMAJ, we analyzed a consecutive cohort of all 44 articles published from July 1, 1996, through August 15, 1997.
For each selected article, the abstract was scrutinized by 1 of 3 examiners who identified each datum or other piece of information in the abstract and then sought to relate it to its source in the body of the article, including tables and figures. Two types of discrepancies were sought: (1) data given differently in the abstract and the body and (2) data given in the abstract but not in the body. If either was identified, the abstract was considered deficient. Discrepancies attributable to rounding were not considered to be deficiencies as long as the rounding was done appropriately, and the rounded value appeared in the abstract and the more detailed value in the body.
The proportions of articles containing deficiencies were compared across journals by  2 analysis. On the basis of normal approximation, 95% confidence intervals (CIs) were calculated for each proportion. We also performed a validation study by randomly selecting (using another computer-generated random number sequence) 7 of each set of 44 articles and having these examined by a second (and different) examiner.
RESULTS
Table 1 contains the proportions of deficient abstracts and 95% CIs for each journal, tabulated considering the abstract as the unit, as well as the types of deficiencies found in the 6 journals. The proportion of deficient abstracts ranged from a low of 18% to a high of 68%. Inconsistency between abstract and body was generally more common than omitted data (ie, data in the abstract not found in the body). A substantial proportion of deficient abstracts contained both kinds of defects (25/104; 24%).
|
|
|
|
Table. Deficient Abstracts*
|
|
|
In the validation study, 38 of the 42 paired comparisons were concordant with respect to identification of deficiencies. The  value for agreement between the 2 evaluators was 0.81 (z = 5.22; P<.001).
COMMENT
The frequency with which we found abstracts to be inaccurate, in the sense of containing information not verifiable in the article's main body (including tables and figures, as well as text) was surprisingly large, ranging from 18% to 68% in the 6 journals surveyed. The more common type of the 2 deficiencies was inconsistency between data in the abstract and those in the body. Giving data or other information in the abstract but not in the body was somewhat less common. These findings are all the more surprising considering that the journals studied are all prominent and highly regarded general medical publications whose editors were founding members of the International Committee of Medical Journal Editors, a respected standard-setting body. These journals have full-time professional staffs who can be presumed to devote a good deal of time and energy to editorial and production processes.
Many of the discrepancies identified were quite minor and not likely to cause serious misinterpretation. For example, 1 abstract2 reported the population to consist of "42 consecutive patients," whereas the body indicated it to be "44 consecutive patients of which 42 agreed to participate." Sometimes, however, discrepancies were more serious; for example, 1 abstract3 gave the estimated 15-year survival as 48%, whereas the body of the text indicated it to be 58%.
The specific question we asked in this study—Can the data and other information in the abstract be verified in the body of the article?—does not seem to have been examined before. Previous studies4-5 of abstract quality generally involved overall or global assessment. Most of the recent literature on abstracts has concerned structured abstracts, introduced in 19876 with the goal of making abstracts more informative. Several investigations7-9 indicated that structured abstracts are actually better in quality, more informative, more readable, and a more efficient use of readers' time. Structured abstracts may well offer all of these advantages, but there is little reason to expect them to reduce the types of deficiencies assessed in this study. Indeed, if structured abstracts are more informative (ie, if they provide more information), they might be more likely to be subject to deficiencies we assessed. In the present study, we could not discern any relationship between various structured formats and the deficiencies assessed.
It is important to acknowledge that we addressed only 1 aspect of abstract accuracy in asking if what is in the abstract is consistent with the body of the article. There is another, at least equally important question: Is the important information in the article found in the abstract? Our study was not designed to address this question.
We found previously1 that providing authors with specific instructions about abstract accuracy when they are revising manuscripts is ineffective in preventing the types of defects assessed in this study. If it is important that abstracts be as accurate as possible—and it can hardly be argued otherwise—and if authors cannot be counted on to provide this level of accuracy, the responsibility must be taken by journals' editorial staffs. As part of the copyediting process, the abstract needs to be scrutinized painstakingly on a line-by-line or even word-by-word basis and each bit of information verified individually and specifically.
AUTHOR INFORMATION
Corresponding Author: Roy M. Pitkin, MD, 10921 Wilshire Blvd, Suite 403, Los Angeles, CA 90024-3908 (e-mail:rpitkin{at}greenjournal.org).
Author Affiliations: Obstetrics & Gynecology, Los Angeles, Calif (Dr Pitkin); Chest, Northbrook, Ill (Ms Branagan); and Department of Preventive Medicine, University of Iowa, Iowa City (Dr Burmeister).
REFERENCES
| |
1. Pitkin RM, Branagan MA. Can the accuracy of abstracts be improved by providing specific instructions? a randomized controlled trial. JAMA. 1998;280:267-269.
FREE FULL TEXT
2. Rothwell PM, McDowell Z, Wong CK, Dorman PJ. Doctors and patients don't agree: cross-sectional study of patients' and doctors' perceptions and assessments of disability in multiple sclerosis. BMJ. 1997;314:1580-1583.
FREE FULL TEXT
3. Socié G, Mary J-Y, de Gramont A, et al , for the French Society of Haematology. Paroxysmal nocturnal haemoglobinuria: long-term follow-up and prognostic factors. Lancet. 1996;348:573-577.
FULL TEXT
|
ISI
| PUBMED
4. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994;121:11-21.
FREE FULL TEXT
5. Froom P, Froom J. Deficiencies in structured medical abstracts. J Clin Epidemiol. 1993;46:591-594.
FULL TEXT
|
ISI
| PUBMED
6. Ad Hoc Working Group for Critical Appraisal of the Medical Literature. A proposal for more informative abstracts of clinical articles. Ann Intern Med. 1987;106:598-604.
FULL TEXT
|
ISI
| PUBMED
7. Comans ML, Overbeke AJ. The structural summary: a tool for reader and author. Ned Tijdschr Geneeskd. 1990;134:2338-2343.
PUBMED
8. Narine L, Yee DX, Einarson TR, Ilersich AL. The quality of journal abstracts for research articles. CMAJ. 1991;144:449-453.
ABSTRACT
9. Taddio A, Pain T, Fassos FF, Boon H, Ilersich AL, Einarson TR. Quality of nonstructured and structured abstracts of original research articles in the British Medical Journal, the Canadian Medical Association Journal and the Journal of the American Medical Association. CMAJ. 1994;150:1611-1615.
ABSTRACT
RELATED LETTER
Effectiveness of a Journal Intervention to Improve Abstract Quality
Roy M. Pitkin, Mary Ann Branagan, and Leon F. Burmeister
JAMA. 2000;283(4):481.
EXTRACT
| FULL TEXT
RELATED ARTICLES
The Need for Concrete Improvement in Abstract Quality
Margaret A. Winker
JAMA. 1999;281(12):1129-1130.
EXTRACT
| FULL TEXT
March 24/31, 1999
JAMA. 1999;281(12):1143-1144.
EXTRACT
| FULL TEXT
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
|
Better reporting of randomized trials in biomedical journal and conference abstracts
Hopewell et al.
Journal of Information Science 2008;34:162-173.
ABSTRACT
An untold story?
Lenzer and Brownlee
BMJ 2008;336:532-534.
FULL TEXT
Extending the CONSORT Statement to Randomized Trials of Nonpharmacologic Treatment: Explanation and Elaboration
Boutron et al.
ANN INTERN MED 2008;148:295-309.
ABSTRACT
| FULL TEXT
Bufala spotting, part one: assessing research papers
Jefferson and Zarra
JRSM 2007;100:38-39.
FULL TEXT
DNA polymorphisms and renal disease: a critical appraisal of studies presented at the annual ERA/EDTA and ASN conferences in 2004 and 2005
Mondry et al.
Nephrol Dial Transplant 2006;21:2775-2779.
ABSTRACT
| FULL TEXT
Transition From Meeting Abstract to Full-length Journal Article for Randomized Controlled Trials
Toma et al.
JAMA 2006;295:1281-1287.
ABSTRACT
| FULL TEXT
Abstracts: problems classified from the user perspective
Montesi and Urdiciain
Journal of Information Science 2005;31:515-526.
ABSTRACT
Keeping up: learning in the workplace
Wyatt and Sullivan
BMJ 2005;331:1129-1132.
FULL TEXT
From conference abstract to full paper: differences between data presented in conferences and journals
Rosmarakis et al.
FASEB J. 2005;19:673-680.
ABSTRACT
| FULL TEXT
Addressing the Limitations of Structured Abstracts
Falagas and Vergidis
ANN INTERN MED 2004;141:576-577.
FULL TEXT
Accuracy of Abstracts for Original Research Articles in Pharmacy Journals
Ward et al.
The Annals of Pharmacotherapy 2004;38:1173-1177.
ABSTRACT
| FULL TEXT
Relation between randomized controlled trials published in leading general medical journals and the global burden of disease
Rochon et al.
CMAJ 2004;170:1673-1677.
ABSTRACT
| FULL TEXT
Quality of Abstracts in 3 Clinical Dermatology Journals
Dupuy et al.
Arch Dermatol 2003;139:589-593.
ABSTRACT
| FULL TEXT
Improving the Clarity of Journal Abstracts in Psychology: The Case for Structure
Hartley
Science Communication 2003;24:366-379.
ABSTRACT
Chapter 6: From Review to Publication
Obstet Gynecol 2003;101:81-99.
FULL TEXT
Effects of Technical Editing in Biomedical Journals: A Systematic Review
Wager and Middleton
JAMA 2002;287:2821-2824.
ABSTRACT
| FULL TEXT
An Observational Study of Orthopaedic Abstracts and Subsequent Full-Text Publications
Bhandari et al.
JBJS 2002;84:615-621.
ABSTRACT
| FULL TEXT
Commentary: Toward systematic reviews in epidemiology
Bracken
Int J Epidemiol 2001;30:954-957.
FULL TEXT
How Accurate Are References in Clinical Chemistry?
Siebers
Clin. Chem. 2001;47:606-607.
FULL TEXT
Data Inconsistencies in Abstracts of Articles in Clinical Chemistry
Siebers
Clin. Chem. 2001;47:149-149.
FULL TEXT
Features of DESCRIPTION META tags in public home pages
Craven
Journal of Information Science 2000;26:303-311.
ABSTRACT
Are structured abstracts more or less accurate than traditional ones? A study in the psychological literature
Hartley
Journal of Information Science 2000;26:273-277.
ABSTRACT
Effectiveness of a Journal Intervention to Improve Abstract Quality
Pitkin et al.
JAMA 2000;283:481-481.
FULL TEXT
|
