Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language

M. Nylenna, P. Riis and Y. Karlsson
Journal of the Norwegian Medical Association, Lysaker.

OBJECTIVE--To study the association between referee characteristics and their manuscript assessments, the influence of manuscript language on referees' judgments, and the usefulness, quality, and extent of referees' free-text comments. DESIGN--Two nonauthentic, but realistic, short manuscripts with a number of common methodological flaws were sent to 180 Scandinavian referees. Through randomization, each referee received one of the manuscripts in English and the other manuscript in the national language. A structured assessment of the manuscript quality was expressed on a 5-point scale, and the impact of referee characteristics (age, gender, experience, and so on) was analyzed by multiple linear regression. MAIN OUTCOME--Manuscript quality assessed by referees. RESULTS--A total of 312 reviews from 156 referees could be used for the study of referee characteristics and language. With increasing experience, the referees gave lower quality scores (P < .05). A tendency toward stricter assessment with younger age was seen (P < .05). No influence of referees' gender, specialty, or nationality was found. For the test manuscript of the poorest quality, the English version was assessed to be better than the national-language version (P < .05). A total of 159 of 312 reviews included free-text comments applicable for the methodological study. In 54 reviews (34%), no methodological comments accompanied the assessment, and in six reviews they were only incomplete. Wrong sampling unit was mentioned by one fourth of 80 referees. Only one referee mentioned the incorrect use of a parametric test in the analysis of data whose distribution was nonparametric. CONCLUSIONS--Experienced and young referees gave a stricter assessment of the manuscripts than their less experienced and older colleagues. An English version seemed to be accepted more easily than a national-language version of the same manuscript. Most referees spontaneously mentioned the shortcomings of the manuscripts only as part of their overall judgment.

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Reviewing the Reviewers: Comparison of Review Quality and Reviewer Characteristics at the American Journal of Roentgenology
Kliewer et al.
Am. J. Roentgenol. 2005;184:1731-1735.
ABSTRACT | FULL TEXT  

Peer Review at the American Journal of Roentgenology: How Reviewer and Manuscript Characteristics Affected Editorial Decisions on 196 Major Papers
Kliewer et al.
Am. J. Roentgenol. 2004;183:1545-1550.
ABSTRACT | FULL TEXT  

Vita Contemplativa: Why I Stopped Trying to Understand the Real World
Starbuck
Organization Studies 2004;25:1233-1254.
ABSTRACT  

Study design, originality and overall consistency influence acceptance or rejection of manuscripts submitted to the Journal: [Le devis de recherche, l'originalite et la coherence generale des manuscrits soumis au Journal influencent leur acceptation ou leur rejet]
Turcotte et al.
Canadian J. Anesthesia 2004;51:549-556.
ABSTRACT | FULL TEXT  

Peer Review: Integral to Science and Indispensable to Annals
Laine et al.
ANN INTERN MED 2003;139:1038-1040.
FULL TEXT  

Turning Lemons into Lemonade: Where Is the Value in Peer Reviews?
Starbuck
Journal of Management Inquiry 2003;12:344-351.
ABSTRACT  

Judging the judges: the role of journal editors
Ray
QJM 2002;95:769-774.
FULL TEXT  

Measuring the Quality of Editorial Peer Review
Jefferson et al.
JAMA 2002;287:2786-2790.
ABSTRACT | FULL TEXT  

Systematic reviews in epidemiology: why are we so far behind?
Dickersin
Int J Epidemiol 2002;31:6-12.
FULL TEXT  

Publications on paediatric anaesthesia: a quantitative analysis of publication activity and international recognition
Brambrink et al.
Br J Anaesth 2000;85:556-562.
ABSTRACT | FULL TEXT  

Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts
Callaham et al.
JAMA 1998;280:229-231.
ABSTRACT | FULL TEXT  

What Makes a Good Reviewer and a Good Review for a General Medical Journal?
Black et al.
JAMA 1998;280:231-233.
ABSTRACT | FULL TEXT  

Effect on the Quality of Peer Review of Blinding Reviewers and Asking Them to Sign Their Reports: A Randomized Controlled Trial
Godlee et al.
JAMA 1998;280:237-240.
ABSTRACT | FULL TEXT  

Unpublished Research From a Medical Specialty Meeting: Why Investigators Fail to Publish
Weber et al.
JAMA 1998;280:257-259.
ABSTRACT | FULL TEXT  

Peer Review for Journals as it Stands Today--Part 1
CAMPANARIO
Science Communication 1998;19:181-211.
ABSTRACT  

Meta-analysis: Unresolved issues and future developments
Smith and Egger
BMJ 1998;316:221-225.
FULL TEXT  

And now, evidence based editing
Smith and Rennie
BMJ 1995;311:826-826.
FULL TEXT  

JSGI: Where We Are I Year Later
Lobo
Reproductive Sciences 1995;2:515-515.