This Article  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (13)  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Topic Collections  • Journalology/ Peer Review/ Authorship  • Alert me on articles by topic  Social Bookmarking   What's this?

Drummond Rennie, MD

JAMA. 1998;280:214-215.

In this issue of THE JOURNAL, we publish 33 articles, based on presentations in Prague, Czech Republic, on subjects surrounding the publication of science relevant to clinicians. This constitutes the third such theme issue of JAMA, since our announcement in 1986 that we would hold a conference to present research into editorial peer review.¹

Publication has always been a central part of the research process, but there had, before this, been remarkably few serious investigations into its workings,^1-3 though no shortage of opinion written in the absence of facts. The lack of information was particularly surprising given the strong prejudices being advanced for and against peer review.¹ Now, 12 years later, some questions about peer review are being answered. More importantly, a growing number of our colleagues accept that their processes are worthy of serious study, and that, as we face the future, it makes sense to monitor our performance and to base changes in the way journals operate on objective evidence.

Kassirer and Campion,⁴ in a challenging presentation given at the second congress in 1993, pointed out that the vast majority of presentations at the 1989 congress dealt with manuscript management , by which they meant "the overall process by which editors manage manuscripts," rather than with manuscript assessment .⁵ Though this is a rather simplistic division (for example, they group articles on the history and philosophy of peer review under "management"), few attempted to get into the minds of the reviewers and examine the cognitive aspects of manuscript assessment. There is no secret about the reason for this: research into the cognitive processes used by peer reviewers, which is essential, is difficult, time-consuming, and expensive. And it is, therefore, no surprise that the articles in this issue of THE JOURNAL also examine the effects of the thinking of authors, reviewers, and editors rather than dissect the mental processes of those involved.

But we should reject the implication that inquiry into all the other aspects of the editorial process is somewhat like photographing a poet at her desk—too removed from the essential activity to be anything but a distraction. Since the results of peer review must depend on those interesting cognitive processes, flaws in the cognitive processes can be identified by studying their consequences. Once identified, these may be corrected and methods to avoid them devised and taught. To take one example: bias is one of the greatest problems facing science. We do not have to know the precise reasons any particular person has an unconscious bias, something that might be revealed by cognitive research, but it is reasonable to suppose that identifying the results of biases, for instance, by showing the existence and extent of bias in favor of publishing positive results, is a powerful and necessary first step in sensitizing authors, reviewers, and editors to this problem and in getting support for preventive measures, such as the registration of trials at inception.^6-7 The 2 previous peer review congresses have featured important articles on bias, including those documenting the unethical failure to complete and publish trials on patients,^6-8 and the Prague congress has followed that tradition.

In this issue of THE JOURNAL, Dickersin, who has been a pioneer in documenting the existence of publication bias, and her colleagues⁹ focus on what seems to be a bias against women in the selection of editors. How these results apply to the selection of manuscripts written by women is still unclear, but the study by Gilbert et al¹⁰ suggested that in 1991 articles submitted to JAMA were not accepted at different rates depending on the sex of the corresponding author and the assigned editor. It is hard to know the significance of Link's¹¹ observation of a bias displayed by reviewers for Gastroenterology in favor of manuscripts from the United States because there was no independent measure of manuscript quality. Looking at another potential bias, Junker¹² found that trials published in German were reported as well (or as badly) as those in English, so there is no justification for not including trials in German in systematic reviews. Unhappily, the trials studied were published before the introduction, in 1996, of the Consolidating Standards of Reporting Trials protocol for reporting trials. Whatever the bias revealed by these studies, Joyce et al¹³ document a distressingly parochial view of science on the part of those who write about such trials in review articles. And, 3 articles^14-16 confirm the pervasive existence of bias in favor of publishing positive results, it being particularly disturbing, but relevant, that acceptance of abstracts for presentation depended on positive outcome rather than quality or design.

Perhaps the issue of bias can be looked at in other ways. Previous work has shown that concerns about bias in editorial peer review may equally apply to peer review of grant applications. Glantz and Bero¹⁷ took advantage of the large numbers of peer reviewers of grant applications, and the fact that their names could be obtained, to examine their qualifications for this task. Wenneras and Wold¹⁸ showed last year in Nature that even in Sweden, a country that prides itself on its social equality, there is heavy bias against women in the peer review of grant applications by the Swedish Medical Research Council.¹⁸ These authors also showed that there was bias in favor of those candidates who knew members of the panels in that extra high marks were awarded in compensation when a panel member recused himself because of a conflict. Wenneras and Wold¹⁸ were assisted by the fact that each candidate was marked in a standardized manner by several reviewers, and they were able to obtain the marks by going to court under the Swedish Freedom of the Press Act. Journal manuscripts usually have several authors, often of different sexes, and are customarily judged in a far less standardized way, so possible biases may be less clear-cut. In addition, editorial peer reviews in the United States are not available through the court system. Given the relative ease with which the grant reviewing process can be studied, it is surprising that more have not looked into this analogous process for the light it could shed on editorial practices. The National Science Foundation has pledged to support such activities, which would follow the pioneering studies of Cole et al,¹⁹ so we should expect more such articles in the future.²⁰

Experience has shown how hard it is to pull together editors at different journals to attack specific questions. Some editors are part-time and specialty, others are full-time and general; all are exceedingly busy. The PEER group took years to assemble, but now the initiative has been given a powerful boost by the editors of BMJ and of the Annals of Emergency Medicine. Though the experience of these groups reinforces the notion that our tools for dissecting the process are blunt, it has confirmed across several journals that studies can be productive. To both editors and authors, the ideal reviewer appears to be young, have been trained in statistics or epidemiology,²¹ and have been rated highly already,²² but even these paragons often missed errors, while editors showed considerable variability in their ratings of the reviewers.^22-23 Reviewers often seemed inconstant, unreliable, and even incompetent.

While this fits in with my own experience, it is possible that more senior reviewers cut more quickly to the chase, concentrating on such ineffable and subjective qualities of a manuscript as its interest to readers or importance to the field. The research instruments may not capture quality well. Editors may have reason to trust the judgment of experienced people, though their summary reviews may seem cursory or even dismissive. The specificity and accuracy of statistical reviews are easier to judge favorably than are such general comments, yet the latter may rightly be given more importance by the editor and so may serve their function at least as well. Whether that is true or not, the editor of BMJ now has to deal with the finding that one of the predictors of a good review was not being a member of an editorial board.²¹ Interventional studies of masking the reviewers to the authors' identities to our surprise showed no useful effect, and the variable success of masking introduced new biases.^23-27 Efforts to improve peer review and the quality of manuscripts also had variable success.^28-32

When Kassirer and Campion⁴ conclude that peer review has been indispensible for the progress of biomedical science, they fall into the category of editors who make assertions in the absence of convincing facts. Having said that, I would add that I agree with them, and so do most of the editors I know. Editors tend to write approvingly about peer review because they believe this formalized expression of criticism to be one of the glories of science. In this, they are merely reflecting the views of the scientific communities they represent. This strong bias in favor of a system that is so time-consuming, complex, expensive, and prone to abuse is itself interesting. It would be nice to find out if it is justified.

We heard about 2 other initiatives in Prague. Clarke and Chalmers³³ examined 26 randomized trials appearing in major journals to assess the extent to which new results were discussed in the light of the totality of available evidence. They found that in 19 trials no such attempt had been made and in only 2 trials were the new results discussed in the context of an updated systematic review of earlier trials. Journals have got a long way to go to abide by standards they may have published themselves.³³ Finally, given the sealed nature of journal processes, The Lancet's appointment of an ombudsperson to handle complaints³⁴ is an original and promising experiment that other journals will watch closely. Perhaps it will encourage journals to be self-critical and researchers to help them to be so. The increasing success of the 3 congresses in luring scientists to look into editorial processes should make us all optimistic.

AUTHOR INFORMATION
Editorials represent the opinions of the authors and The Journal and not those of the American Medical Association

Reprints: Drummond Rennie, MD, JAMA, 515 N State St, Chicago, IL 60610.

Dr Rennie is Deputy Editor (West), JAMA.

REFERENCES
1. Rennie D. Guarding the guardians: a conference on editorial peer review. JAMA. 1986;256:2391-2392. FREE FULL TEXT 2. Bailar JC, Patterson K. Journal peer review: the need for a research agenda. N Engl J Med. 1985;312:654-657. ISI | PUBMED 3. Lock S. A Difficult Balance: Editorial Peer Review in Medicine. London, England: The Nuffield Provincial Hospitals Trust; 1985. 4. Kassirer JP, Campion EW. Peer review: crude and understudied, but indispensible. JAMA. 1994;272:96-97. FREE FULL TEXT 5. Guarding the guardians: research on editorial peer review. JAMA. 1990;263(theme issue):1317-1441. FREE FULL TEXT 6. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:1385-1389. FREE FULL TEXT 7. Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias. JAMA. 1990;263:1392-1395. FREE FULL TEXT 8. Chalmers I. Underreporting research is scientific misconduct. JAMA. 1990;263:1405-1408. FREE FULL TEXT 9. Dickersin K, Fredman L, Flegal KM, Scott JD, Crawley B. Is there a sex bias in choosing editors? epidemiology journals as an example. JAMA. 1998;280:260-264. FREE FULL TEXT 10. Gilbert JR, Williams ES, Lundberg GD. Is there gender bias in JAMA's peer review process? JAMA. 1994;272:139-142. FREE FULL TEXT 11. Link AM. US and non-US submissions: an analysis of reviewer bias. JAMA. 1998;280:246-247. FREE FULL TEXT 12. Junker CA. Adherence to published standards of reporting: a comparison of placebo-controlled trials in English or German. JAMA. 1998;280:247-249. FREE FULL TEXT 13. Joyce J, Rabe-Hesketh S, Wessely S. Reviewing the reviews: the example of chronic fatigue syndrome. JAMA. 1998;280:264-266. FREE FULL TEXT 14. Misakian AL, Bero LA. Publication bias and research on passive smoking: comparison of published and unpublished studies. JAMA. 1998;280:250-253. FREE FULL TEXT 15. Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA. 1998;280:254-257. FREE FULL TEXT 16. Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA. 1998;280:257-259. FREE FULL TEXT 17. Glantz SA, Bero LA. Inappropriate and appropriate selection of "peers" in grant review. JAMA. 1994;272:114-116. FREE FULL TEXT 18. Wenneras C, Wold A. Nepotism and sexism in peer-review. Nature. 1997;387:341-343. FULL TEXT | PUBMED 19. Cole S, Cole R, Simon GA. Chance and consensus in peer review. Science. 1981;214:881-886. FREE FULL TEXT 20. National Science Foundation. National Science Foundation Proposal Advisory Team Report. Arlington, Va: National Science Foundation; 1997. 21. Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA. 1998;280:231-233. FREE FULL TEXT 22. Callaham ML, Baxt WG, Waeckerle JF, Wears RL. Reliability of editors' subjective quality ratings of peer reviews of manuscripts. JAMA. 1998;280:229-231. FREE FULL TEXT 23. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998;280:237-240. FREE FULL TEXT 24. van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA. 1998;280:234-237. FREE FULL TEXT 25. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D, and the PEER Investigators. Does masking author identity improve peer review quality? a randomized controlled trial. JAMA. 1998;280:240-242. FREE FULL TEXT 26. Cho MK, Justice AC, Winker MA, et al. Masking author identity in peer review: what factors influence masking success? JAMA. 1998;280:243-245. FREE FULL TEXT 27. Davidoff F. Masking, blinding, and peer review: the blind leading the blinded. Ann Intern Med. 1998;128:66-68. FREE FULL TEXT 28. Pitkin RM, Branagan MA. Can the accuracy of abstracts be improved by providing specific instructions? a randomized controlled trial. JAMA. 1998;280:267-269. FREE FULL TEXT 29. Scherer RW, Crawley B. Reporting of randomized clinical trial descriptors and use of structured abstracts. JAMA. 1998;280:269-272. FREE FULL TEXT 30. Hatch CL, Goodman SN. Perceived value of providing peer reviewers with abstracts and preprints of related published and unpublished papers. JAMA. 1998;280:273-274. FREE FULL TEXT 31. Jefferson T, Smith R, Yi Y, Drummond M, Pratt M, Gale R. Evaluating the BMJ guidelines on economic submissions: prospective audit of economic submissions to BMJ and The Lancet. JAMA. 1998;280:275-277. FREE FULL TEXT 32. Jadad AR, Cook DJ, Jones A, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998;280:278-280. FREE FULL TEXT 33. Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA. 1998;280:280-282. FREE FULL TEXT 34. Horton R. The journal ombudsperson: a step toward scientific press oversight. JAMA. 1998;280:298-299. FREE FULL TEXT

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Bias in published cost effectiveness studies: systematic review
Bell et al.
BMJ 2006;332:699-703.
ABSTRACT | FULL TEXT  

JADA INDUSTRY ADVISORY BOARD
Yeager
Journal of the American Dental Association 2002;133:1319-1320.
FULL TEXT  

Fourth International Congress on Peer Review in Biomedical Publication.
Rennie
JAMA 2002;287:2759-2760.
FULL TEXT  

Evidence on peer review---scientific quality control or smokescreen?
Goldbeck-Wood
BMJ 1999;318:44-45.
FULL TEXT