You are seeing this message because your Web browser does not support basic Web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.


ABOUT JAMA
Advanced Search

Welcome   | My Account | E-mail Alerts | Access Rights | Sign In


  Vol. 278 No. 19, November 19, 1997 TABLE OF CONTENTS
  JAMA
  •  Online Features
  Original Contributions
 This Article
 •References
 •Full text PDF
 •Send to a friend
 • Save in My Folder
 •Save to citation manager
 •Permissions
 Citing Articles
 •Citation map
 •Citing articles on HighWire
 •Contact me when this article is cited
 Related Content
 •Similar articles in JAMA
 Social Bookmarking
  Add to CiteULike Add to Connotea Add to Del.icio.us Add to Digg Add to Reddit Add to Technorati Add to Twitter What's this?

Peer Review of the Quality of Care

Reliability and Sources of Variability for Outcome and Process Assessments

Maureen A. Smith, MD, MPH; Adam J. Atherly, MA; Robert L. Kane, MD; James T. Pacala, MD, MS

JAMA. 1997;278(19):1573-1578.


Abstract

Context.
—Peer assessments have traditionally been used to judge the quality of care, but a major drawback has been poor interrater reliability.

Objectives.
—To compare the interrater reliability for outcome and process assessments in a population of frail older adults and to identify systematic sources of variability that contribute to poor reliability.

Setting.
—Eight sites participating in a managed care program that integrates acute and long-term care for frail older adults.

Patients.
—A total of 313 frail older adults.

Design.
—Retrospective review of the medical record with 180 charts randomly assigned to 2 geriatricians, 2 geriatric nurse practitioners, or 1 geriatrician and 1 geriatric nurse practitioner and 133 charts randomly assigned to either a geriatrician or a geriatric nurse practitioner.

Main Outcome Measures.
—Interrater reliabilities for structured implicit judgments about process and outcomes for overall care and care for each of 8 tracer conditions (eg, arthritis).

Results.
—Outcome measures had higher interrater reliability than process measures. Five outcome measures achieved fair to good reliability (more than 0.40), while none of the process measures achieved reliabilities more than 0.40. Three factors contributed to poorer reliabilities for process measures: (1) an inability of reviewers to differentiate among cases with respect to the quality of management, (2) systematic bias from individual reviewers, and (3) systematic bias related to the professional training of the reviewer (ie, physician or nurse practitioner).

Conclusions.
—Peer assessments can play an important role in characterizing the quality of care for complex patients with multiple interrelated chronic conditions, but reliability can be poor. Strategies to achieve adequate reliability for these assessments should be applied. These strategies include emphasizing outcomes measurement, providing more structured assessments to identify true differences in patient management, adjusting systematic bias resulting from the individual reviewer and their professional background, and averaging scores from multiple reviewers. Future research on the reliability of peer assessments should focus on improving the ability of process measures to differentiate among cases with respect to the quality of management and on identifying additional sources of systematic bias for both process and outcome measures. Explicit recognition of factors influencing reliability will strengthen efforts to develop sound measures for quality assurance.



Author Affiliations

From the Division of Health Services Research (Drs Smith and Kane and Mr Atherly) and Division of Epidemiology (Dr Smith), School of Public Health, and Department of Family Practice and Community Health (Dr Pacala), University of Minnesota, Minneapolis.


Footnotes

Reprints: Maureen A. Smith, MD, MPH, Division of Epidemiology, University of Minnesota School of Public Health, 1300 S Second St, Suite 300, Minneapolis, MN 55454-1015 (e-mail: smit0286@gold.tc. umn.edu).



Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati   Add to Twitter Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

How safe are our hospitals?
Ramanath and Hendra
Age Ageing 2008;37:243-245.
FULL TEXT  

Under-reporting of deaths to the coroner by doctors: a retrospective review of deaths in two hospitals in Melbourne, Australia
Charles et al.
Int J Qual Health Care 2007;19:232-236.
ABSTRACT | FULL TEXT  

Payment for Quality in End-Stage Renal Disease
Himmelfarb et al.
J. Am. Soc. Nephrol. 2004;15:3263-3269.
FULL TEXT  

Physician Clinical Performance Assessment: Prospects and Barriers
Landon et al.
JAMA 2003;290:1183-1189.
ABSTRACT | FULL TEXT  

Quality Assessment of a Collaborative Approach for Decreasing Drug-Related Morbidity and Achieving Therapeutic Goals
Isetts et al.
Arch Intern Med 2003;163:1813-1820.
ABSTRACT | FULL TEXT  

The Reliability of Assessing the Appropriateness of Requested Diagnostic Tests
Bindels et al.
Med Decis Making 2003;23:31-37.
ABSTRACT  

Bioethics for clinicians: 23. Disclosure of medical error
Hebert et al.
CMAJ 2001;164:509-513.
ABSTRACT | FULL TEXT  

The Unintended Consequences of Measuring Quality on the Quality of Medical Care
Casalino
NEJM 1999;341:1147-1150.
FULL TEXT  

Quality-of-Care Research: Internal Elegance and External Relevance
Starfield
JAMA 1998;280:1006-1008.
FULL TEXT  





HOME | CURRENT ISSUE | PAST ISSUES | TOPIC COLLECTIONS | CME | SUBMIT | SUBSCRIBE | HELP
CONDITIONS OF USE | PRIVACY POLICY | CONTACT US | SITE MAP
 
© 1997 American Medical Association. All Rights Reserved.