 |
|
Comparison Between Impact Factors and Citations in Evidence-Based Practice Guidelines
To the Editor: Impact factors of medical journals are calculated as the total number of current citations of articles published in a journal during the previous 2 calendar years divided by the total number of designated articles published in that journal during the same period.1 Thus, impact factors indicate the annual average number of citations of articles that have appeared in a given journal.
Impact factors are widely regarded as a quality ranking for scientific journals. Concerns have arisen, however, that scientific communities might be overly reliant on impact factors to assess the worth of scientific publications, as these numbers may be artifically inflated in a number of ways.2-4 A related problem is that entire scientific disciplines tend to be evaluated based on average impact factors of their collective journals.5 Despite these concerns, researchers who have published in journals with high impact factors may be more likely to be rewarded by their institutions.
We attempted to validate this interpretation of impact factors by analyzing the bibliographic citations of articles used to support the Guide to Clinical Preventive Services6 by the US Preventive Services Task Force (USPSTF). The USPSTF guidelines are generally thought to reflect the highest level of scientific evidence. Thus, we hypothesized that the guidelines should be supported by a larger number of citations from journals with high impact factors.
Methods
We assessed all references in all 25 chapters of the current version of the USPSTF guidelines and counted the number of references from each journal listed. The journals' impact factors for 2001 were obtained from the Institute for Scientific Information and the Journal of Citation Reports (Science/Social Science; version for 2001).7 Counts were recombined for journals that were renamed.
Results
Among the total 1740 citations in the reference sections of the 25 chapters, 1531 were from scientific journals and 209 were from academic books and official reports. The most cited journal was JAMA with 135 citations, followed by the American Journal of Preventive Medicine (102), BMJ (77), and The Lancet (70). Fifty-six journals had articles cited more than 5 times, comprising a total of 1185 citations. Of these 56 journals, 6 (11%) had an impact factor of more than 10.0; 10 (18%) had an impact factor of 5.0 to 10.0; 11 (20%) had an impact factor of 3.0 to 5.0; and 28 (51%) had impact factors of less than 3.0. Of this latter group, 11 (20%) had impact factors of less than 2.0. Only 7 journals (13%) appeared in the top 100 journals ranked by impact factors (2001). The median impact factor of these 56 journals was 2.76. There was a significant correlation between impact factors and times cited in the USPSTF guidelines (Kendall r = 0.26, P = .005).
Comment
We found that the number of citations by the USPSTF guidelines roughly parallels the impact factors for the respective journals. Journals with low impact factors, however, were also cited frequently as providing important evidence. This finding may reflect the fact that journals that focus on preventive services tend to have lower impact factors than do journals in other scientific disciplines.
Some of the possible domains of impact of journal articles that cannot be measured by impact factors are changes in readers' knowledge, practice, clinical outcomes, funding priorities for research, and prompting of further learning. Overreliance on impact factors may undervalue the unique contributions of individual areas of research. In the field of clinical or preventive medicine, in particular, citation analyses on evidence-based practice guidelines may be a more accurate assessment of the contributions of individual journals and researchers. Although we only assessed the area of preventive health services, we suspect that this general conclusion may extend to other areas of scientific inquiry.
Acknowledgment: We thank Ms Akiko Yoshida for assistance in editing the manuscript.
Takeo Nakayama, MD, PhD
Department of Medical System Informatics
Tsuguya Fukui, MD, PhD
Department of General Medicine and Clinical Epidemiology
Shunichi Fukuhara, MD, DMsc
Department of Healthcare Research
Kyoto University Graduate School of Medicine
Kyoto, Japan
Kiichiro Tsutani, MD, PhD
Department of Pharmacoeconomics
Graduate School of Pharmaceutical Sciences
The University of Tokyo
Tokyo, Japan
Shigeaki Yamazaki, PhD
Department of Library and Information Science
Aichi Shukutoku University
Aichi, Japan
1. Garfield E. Which medical journals have the greatest impact? Ann Intern Med. 1986;105:313-320.
ISI
| PUBMED
2. Seglen PO. Why the impact factor of journals should not be used for evaluating research? BMJ. 1997;314:498-502.
FREE FULL TEXT
3. Smith R. Journal accused of manipulating impact factor. BMJ. 1997;314:463.
4. Hemmingsson A, Mygind T, Skjennald A, Edgren J. Manipulation of impact factors by editors of scientific journals. AJR Am J Roentgenol. 2002;178:767.
FREE FULL TEXT
5. Valdecasas AG, Castroviejo S, Marcus LF. Reliance on the citation index undermines the study of biodiversity. Nature. 2000;403:698.
PUBMED
6. US Preventive Services Task Force. Guide to clinical preventive services, second edition. Available at: http://www.odphp.osophs.dhhs.gov/pubs/guidecps/. Accessed June 25, 2003.
7. ISI Web of Knowledge. Available at: http://www.isinet.com/isi/products/citation/jcr/jcrweb. Accessibility verified (paid subscription) July 24, 2003.
Letters Section Editor: Stephen J. Lurie, MD, PhD, Senior Editor.
JAMA. 2003;290:755-756.
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994-2005) and their Editors' views
Chew et al.
JRSM 2007;100:142-150.
ABSTRACT
| FULL TEXT
|
