 |
|
Fourth International Congress on Peer Review in Biomedical Publication
Drummond Rennie, MD
JAMA. 2002;287:2759-2760.
In this issue of THE JOURNAL, we publish articles selected from those manuscripts submitted to JAMA following presentation at the Fourth International Congress on Peer Review in Biomedical Publication, held in Barcelona, Spain, September 14 to 16, 2001.
On what was, in Spain, the afternoon of September 11, those of us from JAMA, who had arrived in Barcelona a few days early to make final preparations for the Congress, sat in our hotel rooms, staring with horror at images of the collapse of the World Trade Center towers and the devastation at the Pentagon.
Although our careful planning would probably go for naught, it was obvious, since terrorism works best when it most disrupts, that the Congress had to proceed. Yet it soon became apparent that all those from the Americas who were not already in Europe would be at best delayed, and, as events proved, unable to get to Barcelona at all. This meant that the Congress schedule had to be reordered several times a day, as we continuously received word from individuals all over the world, many stuck indefinitely at airports. In the end, 135 of the original 410 who had registered were unable to get to Barcelona, almost all of them from the United States and Canada. However, 40 of 43 presentations from the podium were given as planned, although sometimes by a different presenter on a different day, and 58 of 65 posters were presented.
On the first full day, at noon, the attendees joined the rest of Europe in standing in silence for 3 minutes in honor of the dead and wounded. Thereafter, everyone made a determined effort to remain positive and to hold what turned out to be vigorous scientific discussions. As in previous Congresses, because the aim was simply to present research and discuss it, not draw up position papers nor decide on consensus, equal time was allotted to discussion as to presentation. In the past, this format has had the effect of drawing people together and furthering debate, and so it proved in Barcelona.
These Congresses on peer review began as a response to a call, in 1983, from Bailar and Patterson1 for studies to be done on editorial peer review. There was an abundance of published opinion on peer review, but few empirical studies, and it was obvious that we knew little about one of the central processes of science. Shortly after, Lock, a pioneer in this and other fields, published his important book, A Difficult Balance,2 on what we knew on peer review.
In 1986, we at JAMA invited people to come to a Congress, to be held in 3 years' time, to present the results of their as yet nonexistent research into peer review.3 To our surprise and relief, 50 abstracts were submitted and the first Congress was held in 1989 in Chicago. It generated a good deal of enthusiasm, and was followed by another in 1993 in Chicago, a third in 1997 in Prague, and the fourth in 2001 in Barcelona.
How successful have these meetings been? If we measure success by the number of abstracts, and by the 4 theme issues containing articles resulting from the Congresses,4-6 we can document success by increasing numbers (Figure 1). It seems clear that there were few articles being published on peer review before we started the initiative in 1986, and that now there are about 170 to 200 per year. Indeed, the increasing interest, whether it is due to the Congresses, has clearly extended beyond them because the citations in MEDLINE for non-Congress years is up and remains steady. Figure 1 suggests that a new area of science has been created, and although this research is usually unfunded and performed by individuals with other professional interests, it is gradually beginning to provide a description of editorial peer review and other editorial processes, and some of their consequences.
|
|
|
|
Figure. Increase in MEDLINE Citations on Peer Review Since First Peer Review Congress
The 2 curves indicate the number of references each year in MEDLINE to "peer review" and "publication bias." The 4 bars indicate the numbers of abstracts (50, 110, 160, and 180) received for the successive Congresses.
|
|
|
Once again, in this issue of THE JOURNAL, we publish studies that fail to show any dramatic effect, let alone improvement, brought about by editorial peer review.7 Yet, despite this, it continues to be the experience of editors that peer review is extraordinarily effective, sometimes in saving the reputations of the authors. Why? It makes good sense that editors would want to enlist the services of those more expert in a particular subject than themselves. And there are powerful reasons why editors might wish to spread the responsibility for unfavorable decisions about manuscripts. But there is another important factor. Peer review represents a crucial democratization of the editorial process, incorporating and educating large numbers of the scientific community, and lessening the impression that editorial decisions are arbitrary.8
But these are impressions. Sixteen years after the initiative started, we find ourselves in the peculiar position of believing still more in the virtues of peer review, a system we know to be "time-consuming, complex, expensive and . . . prone to abuse,"9 while we acknowledge that the scientific evidence for its value is meager. Indeed, if the entire peer-review system did not exist but were now to be proposed as a new invention, it would be hard to convince editors looking at the evidence to go through the trouble and expense. This dissonance suggests that we are using the wrong tools to study the wrong factors.
In the editorial accompanying the last theme issue on peer review,9 I noted that the vast majority of studies presented at the Congresses had examined the mechanism or the effects of peer review, rather than the cognitive processes involved. This remains true today. For example, several quantitative studies have shown the existence of various biases, and many important articles published in the previous peer review issues have measured the extent of such biases. Such findings hint at where the process may have gone wrong, but they do not necessarily explain why. Quantitative research is only the first step to understanding the deeper reasons for these biases on the part of authors, editors, and reviewers.
Horton10 is one of those who has begun to carry out important qualitative investigation. In a small study, which Horton hastens to stress is provisional, he questioned all the contributors to a number of studies published in The Lancet, of which he is editor. The replies to his questions showed so much variation that Horton concluded "a research paper rarely represents the full range of opinions of those scientists whose work it claims to report." To improve matters, we need rigorously conducted qualitative studies and cognitive research to examine the reasons for the anomalies in the process, and these will require adequate funding.
I would have hoped by now that more of us researchers would be investigating the thousands of specialty journals, rather than concentrating on "the big 5" general medical journals, although one advantage of that has been that JAMA can continue to publish studies, as we do in this issue, auditing our progress. These tend to show that, while we may have improved our editorial habits, we still have a long way to go.
In the 1986 editorial announcing the first peer review Congress, I noted the appalling standards then prevalent despite the existence of peer review3:
One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.
In the last 16 years, efforts to systematize reviews and improve the reporting of trials and meta-analyses have borne considerable fruit. But an unbiased reader, roaming at random through a medical library, would find in abundance all the problems I described in 1986. This obvious fact makes it all the more surprising that, although many hundreds of editors have attended these Congresses, they represent a fraction of the thousands out there, who pass up participating in the only meeting devoted to the presentation of research into their craft.
Despite the sad coincidence of its opening 2 days after the World Trade Center catastrophe, we felt, and a survey conducted at the time confirmed, that the Congress was a success. For this I am deeply grateful to my colleagues at JAMA, led by Annette Flanagin, who worked around the clock to make it happen, to Richard Smith and his colleagues at BMJ, and to all those who came and brought to the Congress their intelligence, their humor, and their compassion. I am especially grateful to those who tried to come but were prevented by events, yet still sent us their manuscripts.
AUTHOR INFORMATION
Editorials represent the opinions of the authors and THE JOURNAL and not those of the American Medical Association.
Acknowledgment: I thank the Peer Review Congress Advisory Board members for their support and advice as well as the sponsors. Their names are listed on p 2757 of the print journal. The Congress could not have happened without the enthusiastic support and hard work of Cathy DeAngelis, Helga Fritz, Jennifer Reiling, Rosa Miranda, Roxanne Young, Ronna Henry Siegel, Cassio Lynm, and Sharon Kremkau. I also thank Ryo Shohara for assistance with preparation of the figure.
Corresponding Author and Reprints: Drummond Rennie, MD, JAMA, 515 N State St, Chicago, IL 60610.
Author Affiliations: Dr Rennie is Deputy Editor, JAMA, Chicago, Ill.
REFERENCES
|
1. Bailar JC, Patterson K. Journal peer review: the need for a research agenda. N Engl J Med. 1985;312:654-657.
ISI
| PUBMED
2. Lock S. A Difficult Balance: Editorial Peer Review in Medicine. London, England: Nuffield Provincial Hospitals Trust; 1985.
3. Rennie D. Guarding the guardians. JAMA. 1986;256:2391-2392.
FREE FULL TEXT
4. Guarding the guardians. JAMA. 1990;263(theme issue):1317-1441.
5. Second International Congress on Peer Review in Biomedical Publication. JAMA. 1994;272:91.
FREE FULL TEXT
6. Third International Congress on Peer Review in Biomedical Publication. JAMA. 1998;280(theme issue):213-302.
7. Jefferson T, Alderson P, Wager E, Davidoff F. Effects of editorial peer review: a systematic review. JAMA. 2002;287:2784-2786.
FREE FULL TEXT
8. Rennie D. Problems in peer review and fraud. In: Balancing Act: Essays to Honour Stephen Lock. London, England: Keynes Press; 1991.
9. Rennie D. Peer review in Prague. JAMA. 1998;280:214-215.
FREE FULL TEXT
10. Horton R. The hidden research paper. JAMA. 2002;287:2775-2778.
FREE FULL TEXT
 CiteULike  Connotea  Del.icio.us  Digg  Reddit  Technorati
What's this?
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
|
Sixth International Congress on Peer Review and Biomedical Publication, September 2009: Call for Research
Rennie et al.
JAMA 2007;298:2420-2421.
FULL TEXT
At the frontier of biomedical publication: Chicago 2005
Fister
BMJ 2005;331:838-840.
FULL TEXT
Regulatory Parallels to Daubert: Stakeholder Influence, "Sound Science," and the Delayed Adoption of Health-Protective Standards
Neff and Goldman
Am. J. Public Health 2005;95:S81-S91.
ABSTRACT
| FULL TEXT
Travelling but never arriving: reflections of a retiring editor
Smith
BMJ 2004;329:242-244.
FULL TEXT
Prior convictions: bayesian approaches to the analysis and interpretation of clinical megatrials
Diamond and Kaul
J Am Coll Cardiol 2004;43:1929-1939.
ABSTRACT
| FULL TEXT
A gift from anonymous
Edmunds
Ann. Thorac. Surg. 2004;77:10-11.
FULL TEXT
The Fallacy of Double-Blinded Peer Review
Liebeskind et al.
Am. J. Roentgenol. 2003;181
:1422-1423.
FULL TEXT
Our editorial process - some experiences and reflections
Emmelin and Wall
Scand J Public Health 2003;31:161-168.
Fifth International Congress on Peer Review and Biomedical Publication: Call for Research
Rennie et al.
JAMA 2003;289:1438-1438.
FULL TEXT
Fifth international congress on peer review and biomedical publication
Rennie et al.
BMJ 2003;326:563-564.
FULL TEXT
The RSNA Editorial Fellowship: Editorial Fellows' Perspective
Katz et al.
Radiology 2003;226:309-311.
FULL TEXT
How I Review an Original Scientific Article
Hoppin
Am. J. Respir. Crit. Care Med. 2002;166:1019-1023.
FULL TEXT
The Peer Review Process (aka Peer Reviewology)
Yucha
Biol Res Nurs 2002;4:71-72.
|
