This Article  • References  • Full text PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Social Bookmarking   What's this?

Pilot Testing of a Grading Instrument

Irene D. Feurer, MSEd; Gary J. Becker, MD; Daniel Picus, MD; Estella Ramirez; Michael D. Darcy, MD; Marshall E. Hicks, MD

JAMA. 1994;272(2):98-100.

Abstract
Objective.
—To measure the reliability and preliminary validity of a grading instrument for editors to evaluate the quality of peer reviews.

Design.
—The consecutive sample design included 53 reviews of 23 manuscripts. Reviews were systematically assigned to interrater reliability (n=41; power greater than 0.90 to detect a difference of greater than one point) and preliminary criterion-related validity (n=12) subsamples. Content validity was closely examined.

Setting.
—Nonclinical.

Participants.
—Three graders evaluated reliability. One individual examined content validity and two editors tested preliminary criterion-related validity.

Intervention (Instrument).
—Attributes reflecting two basic dimensions, review content and format, were identified and scored (values are possible points/percent contribution): timeliness, 3/21%; grade sheet, 1/7%; etiquette, 1/7%; sectional narratives, 3/21%; citations, 2/14%; narrative summary, 2/14%; and insights, 2/14%. A scoring guide was provided.

Main Outcome Measures.
—Statistical analyses used to test the interrater reliability of the total score included the intraclass correlation coefficient and analysis of variance with the expectation to uphold the null hypothesis. Kendall's coefficient of concordance was used to test preliminary criterion-related validity.

Results.
—The intraclass correlation coefficient was.84 (P<.001) and a lack of difference between mean scores was demonstrated by analysis of variance (P=.46). Content validity was confirmed and preliminary criterion-related validity was indicated (Kendall's coefficient of concordance=.94, P=.038). Conclusions.—The instrument is reliable. Content validation has been completed, and further criterion-related validation is warranted.

(JAMA. 1994;272:98-100)

Author Affiliations
From the Journal of Vascular and Interventional Radiology, Nashville, Tenn (Ms Feurer); Journal of Vascular and Interventional Radiology Editorial Office and the Miami Vascular Institute, Baptist Hospital of Miami, Miami, Fla (Dr Becker and Ms Ramirez); the Mallinckrodt Institute of Radiology, Washington University School of Medicine, St Louis, Mo (Drs Picus, Darcy, and Hicks).

Footnotes
Presented at the Second International Congress on Peer Review in Biomedical Publication, Chicago, Ill, September 9, 1993.

Reprint requests to Journal of Vascular and Interventional Radiology, Miami Vascular Institute, Baptist Hospital of Miami, 8900 N Kendall Dr, Miami, FL 33176 (Dr Becker).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Judging the judges: the role of journal editors
Ray
QJM 2002;95:769-774.
FULL TEXT  

The Peer Review Process (aka Peer Reviewology)
Yucha
Biol Res Nurs 2002;4:71-72.
 

Measuring the Quality of Editorial Peer Review
Jefferson et al.
JAMA 2002;287:2786-2790.
ABSTRACT | FULL TEXT  

Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts
Callaham et al.
JAMA 1998;280:229-231.
ABSTRACT | FULL TEXT