Peer Review in Prague
- Drummond Rennie, MD
In this issue of THE JOURNAL, we publish 33 articles, based on presentations in Prague, Czech Republic, on subjects surrounding the publication of science relevant to clinicians. This constitutes the third such theme issue of JAMA, since our announcement in 1986 that we would hold a conference to present research into editorial peer review.1
Publication has always been a central part of the research process, but there had, before this, been remarkably few serious investigations into its workings,1-3 though no shortage of opinion written in the absence of facts. The lack of information was particularly surprising given the strong prejudices being advanced for and against peer review.1 Now, 12 years later, some questions about peer review are being answered. More importantly, a growing number of our colleagues accept that their processes are worthy of serious study, and that, as we face the future, it makes sense to monitor our performance and to base changes in the way journals operate on objective evidence.
Kassirer and Campion,4 in a challenging presentation given at the second congress in 1993, pointed out that the vast majority of presentations at the 1989 congress dealt with manuscript management , by which they meant "the overall process by which editors manage manuscripts," rather than with manuscript assessment .5 Though this is a rather simplistic division (for example, they group articles on the history and philosophy of peer review under "management"), few attempted to get into the minds of the reviewers and examine the cognitive aspects of manuscript assessment. There is no secret about the reason for this: research into the cognitive processes used by peer reviewers, which is essential, is difficult, time-consuming, and expensive. And it is, therefore, no surprise that the articles in this issue of THE JOURNAL also examine the effects of the thinking of authors, reviewers, and editors rather than dissect the mental processes of those involved.
But we should reject the implication that inquiry into all the other aspects of the editorial process is somewhat like photographing a poet at her desk—too removed from the essential activity to be anything but a distraction. Since the results of peer review must depend on those interesting cognitive processes, flaws in the cognitive processes can be identified by studying their consequences. Once identified, these may be corrected and methods to avoid them devised and taught. To take one example: bias is one of the greatest problems facing science. We do not have to know the precise reasons any particular person has an unconscious bias, something that might be revealed by cognitive research, but it is reasonable to suppose that identifying the results of biases, for instance, by showing the existence and extent of bias in favor of publishing positive results, is a powerful and necessary first step in sensitizing authors, reviewers, and editors to this problem and in getting support for preventive measures, such as the registration of trials at inception.6-7 The 2 previous peer review congresses have featured important articles on bias, including those documenting the unethical failure to complete and publish trials on patients,6-8 and the Prague congress has followed that tradition.
In this issue of THE JOURNAL, Dickersin, who has been a pioneer in documenting the existence of publication bias, and her colleagues9 focus on what seems to be a bias against women in the selection of editors. How these results apply to the selection of manuscripts written by women is still unclear, but the study by Gilbert et al10 suggested that in 1991 articles submitted to JAMA were not accepted at different rates depending on the sex of the corresponding author and the assigned editor. It is hard to know the significance of Link's11 observation of a bias displayed by reviewers for Gastroenterology in favor of manuscripts from the United States because there was no independent measure of manuscript quality. Looking at another potential bias, Junker12 found that trials published in German were reported as well (or as badly) as those in English, so there is no justification for not including trials in German in systematic reviews. Unhappily, the trials studied were published before the introduction, in 1996, of the Consolidating Standards of Reporting Trials protocol for reporting trials. Whatever the bias revealed by these studies, Joyce et al13 document a distressingly parochial view of science on the part of those who write about such trials in review articles. And, 3 articles14-16 confirm the pervasive existence of bias in favor of publishing positive results, it being particularly disturbing, but relevant, that acceptance of abstracts for presentation depended on positive outcome rather than quality or design.
Perhaps the issue of bias can be looked at in other ways. Previous work has shown that concerns about bias in editorial peer review may equally apply to peer review of grant applications. Glantz and Bero17 took advantage of the large numbers of peer reviewers of grant applications, and the fact that their names could be obtained, to examine their qualifications for this task. Wenneras and Wold18 showed last year in Nature that even in Sweden, a country that prides itself on its social equality, there is heavy bias against women in the peer review of grant applications by the Swedish Medical Research Council.18 These authors also showed that there was bias in favor of those candidates who knew members of the panels in that extra high marks were awarded in compensation when a panel member recused himself because of a conflict. Wenneras and Wold18 were assisted by the fact that each candidate was marked in a standardized manner by several reviewers, and they were able to obtain the marks by going to court under the Swedish Freedom of the Press Act. Journal manuscripts usually have several authors, often of different sexes, and are customarily judged in a far less standardized way, so possible biases may be less clear-cut. In addition, editorial peer reviews in the United States are not available through the court system. Given the relative ease with which the grant reviewing process can be studied, it is surprising that more have not looked into this analogous process for the light it could shed on editorial practices. The National Science Foundation has pledged to support such activities, which would follow the pioneering studies of Cole et al,19 so we should expect more such articles in the future.20
Experience has shown how hard it is to pull together editors at different journals to attack specific questions. Some editors are part-time and specialty, others are full-time and general; all are exceedingly busy. The PEER group took years to assemble, but now the initiative has been given a powerful boost by the editors of BMJ and of the Annals of Emergency Medicine. Though the experience of these groups reinforces the notion that our tools for dissecting the process are blunt, it has confirmed across several journals that studies can be productive. To both editors and authors, the ideal reviewer appears to be young, have been trained in statistics or epidemiology,21 and have been rated highly already,22 but even these paragons often missed errors, while editors showed considerable variability in their ratings of the reviewers.22-23 Reviewers often seemed inconstant, unreliable, and even incompetent.
While this fits in with my own experience, it is possible that more senior reviewers cut more quickly to the chase, concentrating on such ineffable and subjective qualities of a manuscript as its interest to readers or importance to the field. The research instruments may not capture quality well. Editors may have reason to trust the judgment of experienced people, though their summary reviews may seem cursory or even dismissive. The specificity and accuracy of statistical reviews are easier to judge favorably than are such general comments, yet the latter may rightly be given more importance by the editor and so may serve their function at least as well. Whether that is true or not, the editor of BMJ now has to deal with the finding that one of the predictors of a good review was not being a member of an editorial board.21 Interventional studies of masking the reviewers to the authors' identities to our surprise showed no useful effect, and the variable success of masking introduced new biases.23-27 Efforts to improve peer review and the quality of manuscripts also had variable success.28-32
When Kassirer and Campion4 conclude that peer review has been indispensible for the progress of biomedical science, they fall into the category of editors who make assertions in the absence of convincing facts. Having said that, I would add that I agree with them, and so do most of the editors I know. Editors tend to write approvingly about peer review because they believe this formalized expression of criticism to be one of the glories of science. In this, they are merely reflecting the views of the scientific communities they represent. This strong bias in favor of a system that is so time-consuming, complex, expensive, and prone to abuse is itself interesting. It would be nice to find out if it is justified.
We heard about 2 other initiatives in Prague. Clarke and Chalmers33 examined 26 randomized trials appearing in major journals to assess the extent to which new results were discussed in the light of the totality of available evidence. They found that in 19 trials no such attempt had been made and in only 2 trials were the new results discussed in the context of an updated systematic review of earlier trials. Journals have got a long way to go to abide by standards they may have published themselves.33 Finally, given the sealed nature of journal processes, The Lancet's appointment of an ombudsperson to handle complaints34 is an original and promising experiment that other journals will watch closely. Perhaps it will encourage journals to be self-critical and researchers to help them to be so. The increasing success of the 3 congresses in luring scientists to look into editorial processes should make us all optimistic.
Acknowledgments
Editorials represent the opinions of the authors and The Journal and not those of the American Medical Association
Reprints: Drummond Rennie, MD, JAMA, 515 N State St, Chicago, IL 60610.
