 |
|
Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts
Michael L. Callaham, MD;
William G. Baxt, MD;
Joseph F. Waeckerle, MD;
Robert L. Wears, MD
JAMA. 1998;280:229-231.
Context.— Quality of reviewers is crucial to journal quality, but there are usually too many for editors to know them all personally. A reliable method of rating them (for education and monitoring) is needed.
Objective.— Whether editors' quality ratings of peer reviewers are reliable and how they compare with other performance measures.
Design.— A 3.5-year prospective observational study.
Setting.— Peer-reviewed journal.
Participants.— All editors and peer reviewers who reviewed at least 3 manuscripts.
Main Outcome Measures.— Reviewer quality ratings, individual reviewer rate of recommendation for acceptance, congruence between reviewer recommendation and editorial decision (decision congruence), and accuracy in reporting flaws in a masked test manuscript.
Interventions.— Editors rated the quality of each review on a subjective 1 to 5 scale.
Results.— A total of 4161 reviews of 973 manuscripts by 395 reviewers were studied. The within-reviewer intraclass correlation was 0.44 (P<.001), indicating that 20% of the variance seen in the review ratings was attributable to the reviewer. Intraclass correlations for editor and manuscript were only 0.24 and 0.12, respectively. Reviewer average quality ratings correlated poorly with the rate of recommendation for acceptance (R=-0.34) and congruence with editorial decision (R=0.26). Among 124 reviewers of the fictitious manuscript, the mean quality rating for each reviewer was modestly correlated with the number of flaws they reported (R=0.53). Highly rated reviewers reported twice as many flaws as poorly rated reviewers.
Conclusions.— Subjective editor ratings of individual reviewers were moderately reliable and correlated with reviewer ability to report manuscript flaws. Individual reviewer rate of recommendation for acceptance and decision congruence might be thought to be markers of a discriminating (ie, high-quality) reviewer, but these variables were poorly correlated with editors' ratings of review quality or the reviewer's ability to detect flaws in a fictitious manuscript. Therefore, they cannot be substituted for actual quality ratings by editors.
From the Division of Emergency Medicine, Department of Medicine, University of California, San Francisco (Dr Callaham); Department of Emergency Medicine, University of Pennsylvania, Philadelphia (Dr Baxt); Department of Emergency Medicine, University of Missouri at Kansas City (Dr Waeckerle); and Department of Emergency Medicine, University of Florida at Jacksonville (Dr Wears).
 CiteULike  Connotea  Del.icio.us  Digg  Reddit  Technorati  Twitter
What's this?
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
|
Editing the JPH
Vetter
J Public Health (Oxf) 2007;29:215-217.
FULL TEXT
The ups and downs of peer review
Benos et al.
Adv. Physiol. Educ. 2007;31:145-152.
ABSTRACT
| FULL TEXT
Reviewing the Reviewers: Comparison of Review Quality and Reviewer Characteristics at the American Journal of Roentgenology
Kliewer et al.
Am. J. Roentgenol. 2005;184:1731-1735.
ABSTRACT
| FULL TEXT
Peer Review at the American Journal of Roentgenology: How Reviewer and Manuscript Characteristics Affected Editorial Decisions on 196 Major Papers
Kliewer et al.
Am. J. Roentgenol. 2004;183:1545-1550.
ABSTRACT
| FULL TEXT
Judging the judges: the role of journal editors
Ray
QJM 2002;95:769-774.
FULL TEXT
Effect of Written Feedback by Editors on Quality of Reviews: Two Randomized Trials
Callaham et al.
JAMA 2002;287:2781-2783.
ABSTRACT
| FULL TEXT
Author Perception of Peer Review: Impact of Review Quality and Acceptance on Satisfaction
Weber et al.
JAMA 2002;287:2790-2793.
ABSTRACT
| FULL TEXT
Peer review of statistics in medical research: the other problem
Bacchetti
BMJ 2002;324:1271-1273.
FULL TEXT
Peer Review in Prague
Rennie
JAMA 1998;280:214-215.
FULL TEXT
|
