This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on ISI (224)  • Contact me when this article is cited  Related Content  • Related articles  • Similar articles in JAMA

A Prospective Validation Study of 3 Methods for Measuring Quality

John W. Peabody, MD, PhD; Jeff Luck, MBA, PhD; Peter Glassman, MBBS, MSc; Timothy R. Dresselhaus, MD, MPH; Martin Lee, PhD

JAMA. 2000;283:1715-1722.

Context  Better health care quality is a universal goal, yet measuring quality has proven to be difficult and problematic. A central problem has been isolating physician practices from other effects of the health care system.

Objective  To validate clinical vignettes as a method for measuring the competence of physicians and the quality of their actual practice.

Design  Prospective trial conducted in 1997 comparing 3 methods for measuring the quality of care for 4 common outpatient conditions: (1) structured reports by standardized patients (SPs), trained actors who presented unannounced to physicians' clinics (the gold standard); (2) abstraction of medical records for those same visits; and (3) physicians' responses to clinical vignettes that exactly corresponded to the SPs' presentations.

Setting  Outpatient primary care clinics at 2 Veterans Affairs medical centers.

Participants  Ninety-eight (97%) of 101 general internal medicine staff physicians, faculty, and second- and third-year residents consented to be randomized for the study. From this group, 10 physicians at each site were randomly selected for inclusion.

Main Outcome Measures  A total of 160 quality scores (8 cases x 20 physicians) were generated for each method using identical explicit criteria based on national guidelines and local expert panels. Scores were defined as the percentage of process criteria correctly met and were compared among the 3 methods.

Results  The quality of care, as measured by all 3 methods, ranged from 76.2% (SPs) to 71.0% (vignettes) to 65.6% (chart abstraction). Measuring quality using vignettes consistently produced scores closer to the gold standard of SP scores than using chart abstraction. This pattern was robust when the scores were disaggregated by the 4 conditions (P<.001 to <.05), by case complexity (P<.001), by site (P<.001), and by level of physician training (P values from <.001 to <.05). The pattern persisted, although less dominantly, when we assessed the component domains of the clinical encounter—history, physical examination, diagnosis, and treatment. Vignettes were responsive to expected directions of variation in quality between sites and levels of training. The vignette responses did not appear to be sensitive to physicians' having seen an SP presenting with the same case.

Conclusions  Our data indicate that quality of health care can be measured in an outpatient setting by using clinical vignettes. Vignettes appear to be a valid and comprehensive method that directly focuses on the process of care provided in actual clinical practice. Vignettes show promise as an inexpensive case-mix adjusted method for measuring the quality of care provided by a group of physicians.

Author Affiliations: San Francisco Veterans Affairs Medical Center and Institute for Global Health, University of California, San Francisco (Dr Peabody); RAND, Santa Monica (Dr Peabody); Veterans Affairs, Greater Los Angeles Healthcare System, West Los Angeles (Drs Peabody, Luck, and Glassman); University of California, Los Angeles, Schools of Medicine and Public Health, Los Angeles (Drs Luck, Peabody, and Lee); Veterans Affairs Center for the Study of Health Care Provider Behavior (Drs Peabody, Luck, Lee, and Glassman); and San Diego Veterans Affairs Medical Center, University of California, San Diego School of Medicine (Dr Dresselhaus).

RELATED ARTICLES

The Quest to Quantify Quality
Stephan D. Fihn
JAMA. 2000;283(13):1740-1742.
EXTRACT | FULL TEXT  

April 5, 2000
JAMA. 2000;283(13):1759-1760.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Individual Physician Practice Variation in Hematopoietic Cell Transplantation
Lee et al.
JCO 2008;26:2162-2170.
ABSTRACT | FULL TEXT  

Discretionary Decision Making By Primary Care Physicians And The Cost Of U.S. Health Care
Sirovich et al.
Health Aff (Millwood) 2008;27:813-823.
ABSTRACT | FULL TEXT  

Reliability and Validity of a Questionnaire to Measure Colorectal Cancer Screening Behaviors: Does Mode of Survey Administration Matter?
Vernon et al.
Cancer Epidemiol. Biomarkers Prev. 2008;17:758-767.
ABSTRACT | FULL TEXT  

Resident Physicians' Life Expectancy Estimates and Colon Cancer Screening Recommendations in Elderly Patients
Lewis et al.
Med Decis Making 2008;28:254-261.
ABSTRACT  

Patient race/ethnicity, age, gender and education are not related to preference for or response to disclosure
Hobgood et al.
Qual Saf Health Care 2008;17:65-70.
ABSTRACT | FULL TEXT  

Physicians and Nurses Focus on Different Aspects of Guidelines When Deciding Whether to Adopt Them: An Application of Conjoint Analysis
Goossens et al.
Med Decis Making 2008;28:138-145.
ABSTRACT  

Differences between physicians in the likelihood of referral and acceptance of elderly patients for dialysis-influence of age and comorbidity
Visser et al.
Nephrol Dial Transplant 2007;22:3255-3261.
ABSTRACT | FULL TEXT  

Cardiologists Use Pulmonary Artery Catheter Information to Make Homogenous Treatment Decisions
Jain et al.
J Intensive Care Med 2007;22:251-256.
ABSTRACT  

A cross-national comparison of the quality of clinical care using vignettes
Peabody and Liu
Health Policy Plan 2007;22:294-302.
ABSTRACT | FULL TEXT  

Physician Self-disclosure in Primary Care Visits: Enough About You, What About Me?
McDaniel et al.
Arch Intern Med 2007;167:1321-1326.
ABSTRACT | FULL TEXT  

Ratings of Physician Communication by Real and Standardized Patients
Fiscella et al.
Ann Fam Med 2007;5:151-158.
ABSTRACT | FULL TEXT  

Grading Gems: Appraising the Quality of Research for Social Work and Social Care
Taylor et al.
Br J Soc Work 2007;37:335-354.
ABSTRACT | FULL TEXT  

Using case vignettes to measure HIV-related stigma among health professionals in China
Li et al.
Int J Epidemiol 2007;36:178-184.
ABSTRACT | FULL TEXT  

Effect of Patient-Specific Ratings vs Conventional Guidelines on Investigation Decisions in Angina: Appropriateness of Referral and Investigation in Angina (ARIA) Trial
Junghans et al.
Arch Intern Med 2007;167:195-202.
ABSTRACT | FULL TEXT  

Factors Associated With Primary Care Clinicians' Choice of a Watchful Waiting Approach to Managing Depression
Meredith et al.
Psychiatr. Serv. 2007;58:72-78.
ABSTRACT | FULL TEXT  

Variation in long-term care needs assessors' willingness to support clients' requests for admission to a residential home: a vignette study.
Jorg et al.
J Aging Health 2006;18:767-790.
ABSTRACT  

The Quality of Obstructive Lung Disease Care for Adults in the United States as Measured by Adherence to Recommended Processes
Mularski et al.
Chest 2006;130:1844-1850.
ABSTRACT | FULL TEXT  

Compliance With American Academy of Pediatrics and American Public Health Association Illness Exclusion Guidelines for Child Care Centers in Maryland: Who Follows Them and When?
Copeland et al.
Pediatrics 2006;118:e1369-e1380.
ABSTRACT | FULL TEXT  

Factorial Surveys: Using Vignettes to Study Professional Judgement
Taylor
Br J Soc Work 2006;36:1187-1207.
ABSTRACT | FULL TEXT  

Predictors of nutrition counseling behaviors and attitudes in US medical students.
Spencer et al.
Am. J. Clin. Nutr. 2006;84:655-662.
ABSTRACT | FULL TEXT  

Optimizing Outcomes in Psychopharmacology Continuing Medical Education (CME): Measuring Learning and Attitudes That May Predict Knowledge Translation into Clinical Practice
Stahl et al.
Focus 2006;4:487.
ABSTRACT | FULL TEXT  

Choosing Your Words Carefully: How Physicians Would Disclose Harmful Medical Errors to Patients.
Gallagher et al.
Arch Intern Med 2006;166:1585-1593.
ABSTRACT | FULL TEXT  

Overprescribing of lipid lowering agents.
Smith et al.
Qual Saf Health Care 2006;15:251-257.
ABSTRACT | FULL TEXT  

Hearing Screening Practices Among a National Sample of Primary Care Pediatricians
Wall et al.
CLIN PEDIATR 2006;45:559-566.
ABSTRACT  

Parental Preferences for Error Disclosure, Reporting, and Legal Action After Medical Error in the Care of Their Children
Hobgood et al.
Pediatrics 2005;116:1276-1286.
ABSTRACT | FULL TEXT  

Variation in the Tendency of Primary Care Physicians to Intervene
Sirovich et al.
Arch Intern Med 2005;165:2252-2256.
ABSTRACT | FULL TEXT  

Managing Febrile Infants: Impact of Literature Recommendations Published During a Physician's Residency
Cox et al.
Eval Health Prof 2005;28:328-348.
ABSTRACT  

Patient-Centered Communication and Diagnostic Testing
Epstein et al.
Ann Fam Med 2005;3:415-421.
ABSTRACT | FULL TEXT  

Is Physician Gender Associated With the Quality of Diabetes Care?
Kim et al.
Diabetes Care 2005;28:1594-1598.
ABSTRACT | FULL TEXT  

Explaining Variation in Physician Practice Patterns and Their Propensities to Recommend Services
O'Neill and Kuder
Med Care Res Rev 2005;62:339-357.
ABSTRACT  

Interrater Reliability: Completing the Methods Description in Medical Records Review Studies
Yawn and Wollan
Am J Epidemiol 2005;161:974-977.
ABSTRACT | FULL TEXT  

Clinical Vignette-Based Surveys: A Tool for Assessing Physician Practice Variation
Veloski et al.
American Journal of Medical Quality 2005;20:151-157.
ABSTRACT  

Influence of Patients' Requests for Direct-to-Consumer Advertised Antidepressants: A Randomized Controlled Trial
Kravitz et al.
JAMA 2005;293:1995-2002.
ABSTRACT | FULL TEXT  

Comparison of Quality of Care for Patients in the Veterans Health Administration and Patients in a National Sample
Asch et al.
ANN INTERN MED 2004;141:938-945.
ABSTRACT | FULL TEXT  

Measuring the Quality of Physician Practice by Using Clinical Vignettes: A Prospective Validation Study
Peabody et al.
ANN INTERN MED 2004;141:771-780.
ABSTRACT | FULL TEXT  

Actors' Portrayals of Depression to Test Interrater Reliability in Clinical Trials
Rosen et al.
Am. J. Psychiatry 2004;161:1909-1911.
ABSTRACT | FULL TEXT  

Using standardised patients in an objective structured clinical examination as a patient safety tool
Battles et al.
Qual Saf Health Care 2004;13:i46-i50.
ABSTRACT | FULL TEXT  

The Primary Care Differential Diagnosis of Inhalational Anthrax
Temte and Zinkel
Ann Fam Med 2004;2:438-444.
ABSTRACT | FULL TEXT  

Diagnosis of Lower-Extremity Deep Vein Thrombosis in Outpatients With Musculoskeletal Disorders: A National Survey Study of Physical Therapists
Riddle et al.
ptjournal 2004;84:717-728.
ABSTRACT | FULL TEXT  

Managed Care Organization and the Quality of Diabetes Care: The Translating Research Into Action for Diabetes (TRIAD) study
Kim et al.
Diabetes Care 2004;27:1529-1534.
ABSTRACT | FULL TEXT  

Migraine practice patterns among neurologists
Lipton et al.
Neurology 2004;62:1926-1931.
ABSTRACT | FULL TEXT  

Profiling The Quality Of Care In Twelve Communities: Results From The CQI Study
Kerr et al.
Health Aff (Millwood) 2004;23:247-256.
ABSTRACT | FULL TEXT  

Avoidance of Anticipated Regret: The Ordering of Prostate-Specific Antigen Tests
Sorum et al.
Med Decis Making 2004;24:149-159.
ABSTRACT  

How Do Primary Care Physicians Manage Children With Possible Developmental Delays? A National Survey With an Experimental Design
Sices et al.
Pediatrics 2004;113:274-282.
ABSTRACT | FULL TEXT  

Depression Management in Medical Clinics: Does Healthcare Sector Make a Difference?
Frayne et al.
American Journal of Medical Quality 2004;19:28-36.
ABSTRACT  

The measurement of active errors: methodological issues
Lilford et al.
Qual Saf Health Care 2003;12:ii8-12.
ABSTRACT | FULL TEXT  

Quality of Health Care Delivered to Adults in the United States
Spiegel et al.
NEJM 2003;349:1866-1868.
FULL TEXT  

Predictors of variation in office visit interval assignment
DeSalvo et al.
Int J Qual Health Care 2003;15:399-405.
ABSTRACT | FULL TEXT  

Physician Clinical Performance Assessment: Prospects and Barriers
Landon et al.
JAMA 2003;290:1183-1189.
ABSTRACT | FULL TEXT  

Management of lower urinary tract infection in women by Slovene GPs
Car et al.
Fam Pract 2003;20:452-456.
ABSTRACT | FULL TEXT  

Why do Primary Care Physicians in the United States and France Order Prostate-Specific Antigen Tests for Asymptomatic Patients?
Sorum et al.
Med Decis Making 2003;23:301-313.
ABSTRACT  

The Quality of Health Care Delivered to Adults in the United States
McGlynn et al.
NEJM 2003;348:2635-2645.
ABSTRACT | FULL TEXT  

Assessing residents' prescribing behavior in renal impairment
SALOMON et al.
Int J Qual Health Care 2003;15:235-240.
ABSTRACT | FULL TEXT  

Using standardized patients to measure professional performance of physicians
BEAULIEU et al.
Int J Qual Health Care 2003;15:251-259.
ABSTRACT | FULL TEXT  

Effects of a Depression Education Program on Residents' Knowledge, Attitudes, and Clinical Skills
Learman et al.
Obstet Gynecol 2003;101:167-174.
ABSTRACT | FULL TEXT  

Bronchiolitis Management Preferences and the Influence of Pulse Oximetry and Respiratory Rate on the Decision to Admit
Mallory et al.
Pediatrics 2003;111:e45-51.
ABSTRACT | FULL TEXT  

The ethical problem of false positives: a prospective evaluation of physician reporting in the medical record
Dresselhaus et al.
J. Med. Ethics 2002;28:291-294.
ABSTRACT | FULL TEXT  

Using standardised patients to measure physicians' practice: validation study using audio recordings
Luck and Peabody
BMJ 2002;325:679-679.
ABSTRACT | FULL TEXT  

Guideline adherence rates and interprofessional variation in a vignette study of depression
Tiemeier et al.
Qual Saf Health Care 2002;11:214-218.
ABSTRACT | FULL TEXT  

Palliative Chemotherapy or Watchful Waiting? A Vignettes Study Among Oncologists
Koedoot et al.
JCO 2002;20:3658-3664.
ABSTRACT | FULL TEXT  

European comparison of costs and quality in the treatment of acute myocardial infarction (2000-2001)
Gandjour et al.
Eur Heart J 2002;23:858-868.
ABSTRACT | FULL TEXT  

Compliance among Pharmacies in California with a Prescription-Drug Discount Program for Medicare Beneficiaries
Lewis et al.
NEJM 2002;346:830-835.
ABSTRACT | FULL TEXT  

Are Psychiatrists' Characteristics Related to How They Care for Depression in the Medically Ill?: Results From a National Case-Vignette Survey
Epstein et al.
Psychosomatics 2001;42:482-489.
ABSTRACT | FULL TEXT  

Invasive procedures carried out in conscious children: contrast between North American and European paediatric oncology centres
Hain et al.
Arch. Dis. Child. 2001;85:12-15.
ABSTRACT | FULL TEXT  

Relation Between Pediatric Experience and Treatment Recommendations for Children and Adolescents With Kidney Failure
Furth et al.
JAMA 2001;285:1027-1033.
ABSTRACT | FULL TEXT  

The Estimated Cost of Managing Focal Spasticity: A Physician Practice Patterns Survey
Radensky et al.
Neurorehabil Neural Repair 2001;15:57-68.
ABSTRACT  

Research and Asthma: Where Do We Go From Here?
Bauchner and Steinbach
Pediatrics 2000;106:897-898.
FULL TEXT  

The Quest to Quantify Quality
Fihn
JAMA 2000;283:1740-1742.
FULL TEXT