Do readers and peer reviewers agree on manuscript quality?
A. C. Justice, J. A. Berlin, S. W. Fletcher, R. H. Fletcher and S. N. Goodman
Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania Medical Center, Philadelphia.
OBJECTIVE--To study readers' judgments of manuscript quality and the degree
to which readers agreed with peer reviewers. DESIGN--Cross-sectional study.
SETTING--Annals of Internal Medicine. SUBJECTS--One hundred thirteen
consecutive manuscripts reporting original research and selected for
publication. Each of two manuscript versions (one before and one after
revision) was judged by two readers, randomly sampled from those who said
(based on the title) that they would read the article; one peer reviewer
(peer), chosen in the usual way for Annals; and one expert in clinical
research methods (expert). Each judge completed an instrument that included
a 10-point subjective summary grade of manuscript quality. MAIN OUTCOME
MEASURES--Agreement on the 10-point summary grade of manuscript quality
between reader-expert, reader-peer, and reader-reader. RESULTS--Readers and
peers gave high grades (77% and 73% gave a grade of 5 or better,
respectively), while experts were more critical (52% gave a grade of 5 or
better; P < .0001). Agreement was relatively high among judge groups (in
all cases, > 69%) but agreement beyond chance was poor (kappa <
0.04). One third of readers (33%) thought that the manuscript had little
relevance to their work. CONCLUSION--Readers, like most peer reviewers, are
generally satisfied with the quality of manuscripts but would like research
articles to be more relevant to their clinical practice.
