Watching the doctor-watchers. How well do peer review organization methods detect hospital care quality problems?

H. R. Rubin, W. H. Rogers, K. L. Kahn, L. V. Rubenstein and R. H. Brook
Department of Medicine, Johns Hopkins University, Baltimore, Md.

OBJECTIVE--To determine how well one state's peer review organization (PRO) judged the quality of hospital care compared with an independent, credible judgment of quality of care. DESIGN--Retrospective study comparing a PRO's review, including initial screening, physician review, and final judgments, with an independent "study judgment" based on blinded, structured, implicit reviews of hospital records. SETTING--One state's medical and surgical Medicare hospitalizations during 1985 through 1987 audited randomly by the state's PRO. SAMPLE--Stratified random sampling of records: 62 records that passed the PRO initial screening process and were not referred for PRO physician review; 50 records that failed the PRO screen and then were confirmed by PRO physicians to be "quality problems." INTERVENTION--None. MAIN OUTCOME MEASURE--A study judgment of below standard or standard or above based on the mean of overall ratings by five internists for records in medical diagnosis related groups (DRGs) and by five internists and five surgeons for surgical DRGs. Each step in the PRO review was evaluated for how many records passing or failing that step were judged standard or above or below standard in the study (positive and negative predictive value) and how well that step classified records that the study judged below standard or standard or above (sensitivity and specificity). RESULTS--An estimated 18% of records reviewed by the PRO were below standard according to the study judgment, compared with 6.3% quality problems according to the PRO's final judgment (difference, 12%; 95% confidence interval, 1 to 23). The PRO's initial screening process failed to detect and refer for PRO physician review two of three records that the study judged below standard. In addition, only one of three of the records that PRO physicians judged to be quality problems were judged below standard by the study judgment. Therefore, the PRO's final quality of care judgment and the study judgment agreed little more than expected by chance, especially about poor quality of care. Although the PRO correctly classified 95% of the records that the study judged standard or above, it detected only 11% of records judged below standard by the study. CONCLUSIONS--Most of all, this PRO review process would be improved by additional preliminary screens to identify the 67% of records that the study judged below standard but that passed its initial screening. The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through improved reviewer selection and training, a structured review method, and more physician reviewers per record.

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Relationship Between Preventability of Death After Coronary Artery Bypass Graft Surgery and All-Cause Risk-Adjusted Mortality Rates
Guru et al.
Circulation 2008;117:2969-2976.
ABSTRACT | FULL TEXT  

Do Quality Improvement Organizations Improve the Quality of Hospital Care for Medicare Beneficiaries?
Snyder and Anderson
JAMA 2005;293:2900-2907.
ABSTRACT | FULL TEXT  

Pursuing integration of performance measures into electronic medical records: beta-adrenergic receptor antagonist medications
Weiner et al.
Qual Saf Health Care 2005;14:99-106.
ABSTRACT | FULL TEXT  

A new instrument to measure appropriateness of services in primary care
Thom et al.
Int J Qual Health Care 2004;16:133-140.
ABSTRACT | FULL TEXT  

Electronically Screening Discharge Summaries for Adverse Medical Events
Murff et al.
J. Am. Med. Inform. Assoc. 2003;10:339-350.
ABSTRACT | FULL TEXT  

Options for Assessing PPO Quality: Accreditation and Profiling as Accountability Strategies
Scanlon et al.
Med Care Res Rev 2001;58:70-100.
ABSTRACT  

Estimating Hospital Deaths Due to Medical Errors: Preventability Is in the Eye of the Reviewer
Hayward and Hofer
JAMA 2001;286:415-420.
ABSTRACT | FULL TEXT  

Bioethics for clinicians: 23. Disclosure of medical error
Hebert et al.
CMAJ 2001;164:509-513.
ABSTRACT | FULL TEXT  

Comparison of Vignettes, Standardized Patients, and Chart Abstraction: A Prospective Validation Study of 3 Methods for Measuring Quality
Peabody et al.
JAMA 2000;283:1715-1722.
ABSTRACT | FULL TEXT  

Hospital Peer Review and the National Practitioner Data Bank: Clinical Privileges Action Reports
Baldwin et al.
JAMA 1999;282:349-355.
ABSTRACT | FULL TEXT  

The Movement for Improved Quality in Health Care
Bodenheimer
NEJM 1999;340:488-492.
FULL TEXT  

Quality Improvement in Health Care: A Brief History of the Medicare Peer Review Organization (PRO) Initiative
Weinmann
Eval Health Prof 1998;21:413-418.
ABSTRACT  

Improving Appropriate Selection of Radical Prostatectomy for Prostate Cancer
Lewis and Rising
Eval Health Prof 1998;21:429-441.
ABSTRACT  

Quality Improvement in the Diagnosis and Treatment of Heart Failure by Participating Indiana and Kentucky Hospitals
Caldwell et al.
Eval Health Prof 1998;21:461-471.
ABSTRACT  

Research Evidence on the Validity of Risk-Adjusted Mortality Rate as a Measure of Hospital Quality of Care
Thomas and Hofer
Med Care Res Rev 1998;55:371-404.
ABSTRACT  

Chart Review: A Need for Reappraisal
Wu and Ashton
Eval Health Prof 1997;20:146-163.
ABSTRACT  

Peer Review Organizations: Scientific Challenges in HCFA's Health Care Quality Improvement Initiative
Hayes et al.
Med Care Res Rev 1994;51:39-60.
 

The Reliability of Peer Assessments: A Meta-Analysis
Goldman
Eval Health Prof 1994;17:3-21.
ABSTRACT  

Can Administrative Data be Used to Compare the Quality of Health Care?
Romano
Med Care Res Rev 1993;50:451-477.
 

PROs and the Health Care Quality Improvement Initiative: Insights from 50 Cases of Serious Medical Mistakes
Feldman and Rundall
Med Care Res Rev 1993;50:123-152.