This Article  • References  • Full text PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Social Bookmarking   What's this?

How Well Do Peer Review Organization Methods Detect Hospital Care Quality Problems?

Haya R. Rubin, MD, PhD; William H. Rogers, PhD; Katherine L. Kahn, MD; Lisa V. Rubenstein, MD; Robert H. Brook, MD, ScD

JAMA. 1992;267(17):2349-2354.

Abstract
Objective.
—To determine how well one state's peer review organization (PRO) judged the quality of hospital care compared with an independent, credible judgment of quality of care.

Design.
—Retrospective study comparing a PRO's review, including initial screening, physician review, and final judgments, with an independent "study judgment" based on blinded, structured, implicit reviews of hospital records.

Setting.
—One state's medical and surgical Medicare hospitalizations during 1985 through 1987 audited randomly by the state's PRO.

Sample.
—Stratified random sampling of records: 62 records that passed the PRO initial screening process and were not referred for PRO physician review; 50 records that failed PRO screen and then were confirmed by PRO physicians to be "quality problems."

Intervention.
—None.

Main Outcome Measure.
—A study judgment of below standard or standard or above based on the mean of overall ratings by five internists for records in medical diagnosis related groups (DRGs) and by five internists and five surgeons for surgical DRGs. Each step in the PRO review was evaluated for how many records passing or failing that step were judged standard or above or below standard in the study (positive and negative predictive value) and how well that step classified records that the study judged below standard or standard or above (sensitivity and specificity).

Results.
—An estimated 18% of records reviewed by the PRO were below standard according to the study judgment, compared with 6.3% quality problems according to the PRO's final judgment (difference, 12%; 95% confidence interval, 1 to 23). The PRO's initial screening process failed to detect and refer for PRO physician review two of three records that the study judged below standard. In addition, only one of three of the records that PRO physicians judged to be quality problems were judged below standard by the study judgment. Therefore, the PRO's final quality of care judgment and the study judgment agreed little more than expected by chance, especially about poor quality of care. Although the PRO correctly classified 95% of the records that the study judged standard or above, it detected only 11% of records judged below standard by the study.

Conclusions.
—Most of all, this PRO review process would be improved by additional preliminary screens to identify the 67% of records that the study judged below standard but that passed its initial screening. The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through improved reviewer selection and training, a structured review method, and more physician reviewers per record.

(JAMA. 1992;267:2349-2354)

Author Affiliations
From the Departments of Medicine and Health Policy and Management, The Johns Hopkins University, The Johns Hopkins Program for Medical Technology and Practice Assessment, and The Johns Hopkins Hospital, Baltimore, Md (Dr Rubin); RAND, Santa Monica, Calif (Drs Rogers, Kahn, Rubenstein, and Brook); UCLA Department of Medicine, Los Angeles, Calif (Drs Kahn and Brook); and the Veterans Administration Medical Center, Sepulveda, Calif (Dr Rubenstein).

Footnotes
The interpretations of the data we report are our own and do not reflect the opinions of any of the sponsors or funding agencies.

Reprint requests to Director, Quality of Care Research, The Johns Hopkins Hospital, 1830 E Monument St, Baltimore, MD 21205 (Dr Rubin).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Relationship Between Preventability of Death After Coronary Artery Bypass Graft Surgery and All-Cause Risk-Adjusted Mortality Rates
Guru et al.
Circulation 2008;117:2969-2976.
ABSTRACT | FULL TEXT  

Do Quality Improvement Organizations Improve the Quality of Hospital Care for Medicare Beneficiaries?
Snyder and Anderson
JAMA 2005;293:2900-2907.
ABSTRACT | FULL TEXT  

Pursuing integration of performance measures into electronic medical records: beta-adrenergic receptor antagonist medications
Weiner et al.
Qual Saf Health Care 2005;14:99-106.
ABSTRACT | FULL TEXT  

A new instrument to measure appropriateness of services in primary care
Thom et al.
Int J Qual Health Care 2004;16:133-140.
ABSTRACT | FULL TEXT  

Electronically Screening Discharge Summaries for Adverse Medical Events
Murff et al.
J. Am. Med. Inform. Assoc. 2003;10:339-350.
ABSTRACT | FULL TEXT  

Options for Assessing PPO Quality: Accreditation and Profiling as Accountability Strategies
Scanlon et al.
Med Care Res Rev 2001;58:70-100.
ABSTRACT  

Estimating Hospital Deaths Due to Medical Errors: Preventability Is in the Eye of the Reviewer
Hayward and Hofer
JAMA 2001;286:415-420.
ABSTRACT | FULL TEXT  

Bioethics for clinicians: 23. Disclosure of medical error
Hebert et al.
CMAJ 2001;164:509-513.
ABSTRACT | FULL TEXT  

Comparison of Vignettes, Standardized Patients, and Chart Abstraction: A Prospective Validation Study of 3 Methods for Measuring Quality
Peabody et al.
JAMA 2000;283:1715-1722.
ABSTRACT | FULL TEXT  

Hospital Peer Review and the National Practitioner Data Bank: Clinical Privileges Action Reports
Baldwin et al.
JAMA 1999;282:349-355.
ABSTRACT | FULL TEXT  

The Movement for Improved Quality in Health Care
Bodenheimer
NEJM 1999;340:488-492.
FULL TEXT  

Quality Improvement in Health Care: A Brief History of the Medicare Peer Review Organization (PRO) Initiative
Weinmann
Eval Health Prof 1998;21:413-418.
ABSTRACT  

Improving Appropriate Selection of Radical Prostatectomy for Prostate Cancer
Lewis and Rising
Eval Health Prof 1998;21:429-441.
ABSTRACT  

Quality Improvement in the Diagnosis and Treatment of Heart Failure by Participating Indiana and Kentucky Hospitals
Caldwell et al.
Eval Health Prof 1998;21:461-471.
ABSTRACT  

Research Evidence on the Validity of Risk-Adjusted Mortality Rate as a Measure of Hospital Quality of Care
Thomas and Hofer
Med Care Res Rev 1998;55:371-404.
ABSTRACT  

Adherence to Prescribed Explicit Criteria During Utilization Review: An Analysis of Communications Between Attending and Reviewing Physicians
Kleinman et al.
JAMA 1997;278:497-501.
ABSTRACT  

Chart Review: A Need for Reappraisal
Wu and Ashton
Eval Health Prof 1997;20:146-163.
ABSTRACT  

Physicians' Mistakes: Will Your Colleagues Offer Support?
Ely
Arch Fam Med 1996;5:76-77.
ABSTRACT  

Quality of Care for Medicare Patients With Acute Myocardial Infarction: A Four-State Pilot Study From the Cooperative Cardiovascular Project
Ellerbeck et al.
JAMA 1995;273:1509-1514.
ABSTRACT  

Assessing the Quality of Care for Children: Prospects Under Health Reform
McGlynn et al.
Arch Pediatr Adolesc Med 1995;149:359-368.
ABSTRACT  

Health System Reform: Forward or Backward With Quality Oversight?
Jost
JAMA 1994;271:1508-1511.
ABSTRACT  

Peer Review Organizations: Scientific Challenges in HCFA's Health Care Quality Improvement Initiative
Hayes et al.
Med Care Res Rev 1994;51:39-60.
 

The Reliability of Peer Assessments: A Meta-Analysis
Goldman
Eval Health Prof 1994;17:3-21.
ABSTRACT  

Can Administrative Data be Used to Compare the Quality of Health Care?
Romano
Med Care Res Rev 1993;50:451-477.
 

Hysterectomy, HMOs, and Appropriateness-Reply
McGlynn et al.
JAMA 1993;270:1317-1318.
ABSTRACT  

Quality Health Care
Laffel and Berwick
JAMA 1993;270:254-255.
ABSTRACT  

PROs and the Health Care Quality Improvement Initiative: Insights from 50 Cases of Serious Medical Mistakes
Feldman and Rundall
Med Care Res Rev 1993;50:123-152.
 

Quality Assessment of Quality Improvement Programs
Overfield
JAMA 1992;268:2648-2648.
ABSTRACT  

The Health Care Quality Improvement Initiative: A New Approach to Quality Assurance in Medicare
Jencks and Wilensky
JAMA 1992;268:900-903.
ABSTRACT  

Is the Quality Cart Before the Horse?
Nash
JAMA 1992;268:917-918.
ABSTRACT