Peer review of the quality of care. Reliability and sources of variability for outcome and process assessments
M. A. Smith, A. J. Atherly, R. L. Kane and J. T. Pacala
Division of Health Services Research, School of Public Health, University of Minnesota, Minneapolis 55454-1015, USA. smit0286@gold.tc.umn.edu
CONTEXT: Peer assessments have traditionally been used to judge the quality
of care, but a major drawback has been poor interrater reliability.
OBJECTIVES: To compare the interrater reliability for outcome and process
assessments in a population of frail older adults and to identify
systematic sources of variability that contribute to poor reliability.
SETTING: Eight sites participating in a managed care program that
integrates acute and long-term care for frail older adults. PATIENTS: A
total of 313 frail older adults. DESIGN: Retrospective review of the
medical record with 180 charts randomly assigned to 2 geriatricians, 2
geriatric nurse practitioners, or 1 geriatrician and 1 geriatric nurse
practitioner and 133 charts randomly assigned to either a geriatrician or a
geriatric nurse practitioner. MAIN OUTCOME MEASURES: Interrater
reliabilities for structured implicit judgments about process and outcomes
for overall care and care for each of 8 tracer conditions (eg, arthritis).
RESULTS: Outcome measures had higher interrater reliability than process
measures. Five outcome measures achieved fair to good reliability (more
than 0.40), while none of the process measures achieved reliabilities more
than 0.40. Three factors contributed to poorer reliabilities for process
measures: (1) an inability of reviewers to differentiate among cases with
respect to the quality of management, (2) systematic bias from individual
reviewers, and (3) systematic bias related to the professional training of
the reviewer (ie, physician or nurse practitioner). CONCLUSIONS: Peer
assessments can play an important role in characterizing the quality of
care for complex patients with multiple interrelated chronic conditions,
but reliability can be poor. Strategies to achieve adequate reliability for
these assessments should be applied. These strategies include emphasizing
outcomes measurement, providing more structured assessments to identify
true differences in patient management, adjusting systematic bias resulting
from the individual reviewer and their professional background, and
averaging scores from multiple reviewers. Future research on the
reliability of peer assessments should focus on improving the ability of
process measures to differentiate among cases with respect to the quality
of management and on identifying additional sources of systematic bias for
both process and outcome measures. Explicit recognition of factors
influencing reliability will strengthen efforts to develop sound measures
for quality assurance.
How safe are our hospitals?
Ramanath and Hendra
Age Ageing 2008;37:243-245.
FULL TEXT
Under-reporting of deaths to the coroner by doctors: a retrospective review of deaths in two hospitals in Melbourne, Australia
Charles et al.
Int J Qual Health Care 2007;19:232-236.
ABSTRACT
| FULL TEXT
Payment for Quality in End-Stage Renal Disease
Himmelfarb et al.
J. Am. Soc. Nephrol. 2004;15:3263-3269.
FULL TEXT
Physician Clinical Performance Assessment: Prospects and Barriers
Landon et al.
JAMA 2003;290:1183-1189.
ABSTRACT
| FULL TEXT
Quality Assessment of a Collaborative Approach for Decreasing Drug-Related Morbidity and Achieving Therapeutic Goals
Isetts et al.
Arch Intern Med 2003;163:1813-1820.
ABSTRACT
| FULL TEXT
The Reliability of Assessing the Appropriateness of Requested Diagnostic Tests
Bindels et al.
Med Decis Making 2003;23:31-37.
ABSTRACT
Bioethics for clinicians: 23. Disclosure of medical error
Hebert et al.
CMAJ 2001;164:509-513.
ABSTRACT
| FULL TEXT
The Unintended Consequences of Measuring Quality on the Quality of Medical Care
Casalino
NEJM 1999;341:1147-1150.
FULL TEXT
Quality-of-Care Research: Internal Elegance and External Relevance
Starfield
JAMA 1998;280:1006-1008.
FULL TEXT
