This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on ISI (184)  • Contact me when this article is cited  Related Content  • Related letter  • Related articles  • Similar articles in JAMA  Topic Collections  • Quality of Care, Other  • Alert me on articles by topic

Timothy P. Hofer, MD, MS; Rodney A. Hayward, MD; Sheldon Greenfield, MD; Edward H. Wagner, MD; Sherrie H. Kaplan, PhD; Willard G. Manning, PhD

JAMA. 1999;281:2098-2105.

Context  Physician profiling is widely used by many health care systems, but little is known about the reliability of commonly used profiling systems.

Objectives  To determine the reliability of a set of physician performance measures for diabetes care, one of the most common conditions in medical practice, and to examine whether physicians could substantially improve their profiles by preferential patient selection.

Design and Setting  Cohort study performed from 1990 to 1993 at 3 geographically and organizationally diverse sites, including a large staff-model health maintenance organization, an urban university teaching clinic, and a group of private-practice physicians in an urban area.

Participants  A total of 3642 patients with type 2 diabetes cared for by 232 different physicians.

Main Outcome Measures  Physician profiles for their patients' hospitalization and clinic visit rates, total laboratory resource utilization rate and level of glycemic control by average hemoglobin A_1c level with and without detailed case-mix adjustment.

Results  For profiles based on hospitalization rates, visit rates, laboratory utilization rates, and glycemic control, 4% or less of the overall variance was attributable to differences in physician practice and the reliability of the median physician's case-mix–adjusted profile was never better than 0.40. At this low level of physician effect, a physician would need to have more than 100 patients with diabetes in a panel for profiles to have a reliability of 0.80 or better (while more than 90% of all primary care physicians at the health maintenance organization had fewer than 60 patients with diabetes). For profiles of glycemic control, high outlier physicians could dramatically improve their physician profile simply by pruning from their panel the 1 to 3 patients with the highest hemoglobin A_1clevels during the prior year. This advantage from gaming could not be prevented by even detailed case-mix adjustment.

Conclusions  Physician "report cards" for diabetes, one of the highest-prevalence conditions in medical practice, were unable to detect reliably true practice differences within the 3 sites studied. Use of individual physician profiles may foster an environment in which physicians can most easily avoid being penalized by avoiding or deselecting patients with high prior cost, poor adherence, or response to treatments.

Author Affiliations: Veterans Affairs Center for Practice Management and Outcomes Research, and Department of Internal Medicine, University of Michigan, Ann Arbor (Drs Hofer and Hayward); Primary Care Outcomes Research Institute, New England Medical Center, Boston, Mass (Drs Greenfield and Kaplan); Center for Health Studies, Group Health Cooperative, Seattle, Wash (Dr Wagner); and Department of Health Studies, University of Chicago, Chicago, Ill (Dr Manning).

RELATED LETTER

Unreliability of Physician "Report Cards" to Assess Cost and Quality of Care
Edwin D. Huff, Lewis G. Sandy, Kaveh G. Shojania, Robert M. Wachter, John E. Wennberg, Timothy P. Hofer, Rodney A. Hayward, Sheldon Greenfield, Sherrie H. Kaplan, Edward H. Wagner, Willard G. Manning, and Andrew B. Bindman
JAMA. 2000;283(1):51-54.
EXTRACT | FULL TEXT  

RELATED ARTICLES

Can Physician Profiles Be Trusted?
Andrew B. Bindman
JAMA. 1999;281(22):2142-2143.
EXTRACT | FULL TEXT  

June 9, 1999
JAMA. 1999;281(22):2157-2158.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Estimating Physician Effects on Glycemic Control in the Treatment of Diabetes: Methods, effects sizes, and implications for treatment policy
Tuerk et al.
Diabetes Care 2008;31:869-873.
ABSTRACT | FULL TEXT  

Performance Measurement in the Small Office Practice: Challenges and Potential Solutions
Landon and Normand
ANN INTERN MED 2008;148:353-357.
ABSTRACT | FULL TEXT  

Performance in Practice: Sample Tools for the Care of Patients with Major Depressive Disorder
Fochtmann et al.
Focus 2008;6:22-35.
ABSTRACT | FULL TEXT  

Risk Stratification and Comorbidity
Ferraris et al.
Card Surg Adult 2008;3:199-246.
FULL TEXT  

Making performance indicators work: experiences of US Veterans Health Administration
Kerr and Fleming
BMJ 2007;335:971-973.
FULL TEXT  

Pay for Performance, Public Reporting, and Racial Disparities in Health Care: How Are Programs Being Designed?
Chien et al.
Med Care Res Rev 2007;64:283S-304S.
ABSTRACT  

Evaluating Medical Training Programs by the Quality of Care Delivered by Their Alumni
Asch et al.
JAMA 2007;298:1049-1051.
FULL TEXT  

Publishing 'quality' measures: how it works and when it does not?
Hamblin
Int J Qual Health Care 2007;19:183-186.
FULL TEXT  

Pay-for-Performance in Pediatrics: Proceed With Caution
Chien and Dudley
Pediatrics 2007;120:186-188.
FULL TEXT  

Will Pay-For-Performance And Quality Reporting Affect Health Care Disparities?
Casalino et al.
Health Aff (Millwood) 2007;26:w405-w414.
ABSTRACT | FULL TEXT  

Performance Measurement in Search of a Path
Hayward
NEJM 2007;356:951-953.
FULL TEXT  

Clinical Concerns About Clinical Performance Measurement
Werner and Asch
Ann Fam Med 2007;5:159-163.
ABSTRACT | FULL TEXT  

General Internists' Views On Pay-For-Performance And Public Reporting Of Quality Scores: A National Survey
Casalino et al.
Health Aff (Millwood) 2007;26:492-499.
ABSTRACT | FULL TEXT  

Advancing Evidence-Based Care For Diabetes: Lessons From The Veterans Health Administration
Kupersmith et al.
Health Aff (Millwood) 2007;26:w156-w168.
ABSTRACT | FULL TEXT  

Impact of Self-Reported Patient Characteristics Upon Assessment of Glycemic Control in the Veterans Health Administration
Maney et al.
Diabetes Care 2007;30:245-251.
ABSTRACT | FULL TEXT  

Summary measures of quality of diabetes care: comparison of continuous weighted performance measurement and dichotomous thresholds
Aron et al.
Int J Qual Health Care 2007;19:29-36.
ABSTRACT | FULL TEXT  

Assessing the Accuracy of Hospital Clinical Performance Measures
Normand et al.
Med Decis Making 2007;27:9-20.
ABSTRACT  

Transferring Critically Ill Patients Out of Hospital Improves the Standardized Mortality Ratio: A Simulation Study
Kahn et al.
Chest 2007;131:68-75.
ABSTRACT | FULL TEXT  

Creating Accountable Care Organizations: The Extended Hospital Medical Staff
Fisher et al.
Health Aff (Millwood) 2007;26:w44-w57.
ABSTRACT | FULL TEXT  

Relationship between quality of care and racial disparities in medicare health plans.
Trivedi et al.
JAMA 2006;296:1998-2004.
ABSTRACT | FULL TEXT  

Is the Number of Documented Diabetes Process-of-Care Indicators Associated With Cardiometabolic Risk Factor Levels, Patient Satisfaction, or Self-Rated Quality of Diabetes Care?: The Translating Research into Action for Diabetes (TRIAD) study.
Ackermann et al.
Diabetes Care 2006;29:2108-2113.
ABSTRACT | FULL TEXT  

Are claims data accurate enough to identify patients for performance measures or quality improvement? The case of diabetes, heart disease, and depression.
Solberg et al.
American Journal of Medical Quality 2006;21:238-245.
ABSTRACT  

Use of a Glucose Algorithm to Direct Diabetes Therapy Improves A1C Outcomes and Defines an Approach to Assess Provider Behavior.
Miller et al.
The Diabetes Educator 2006;32:533-545.
ABSTRACT | FULL TEXT  

Determinants of androgen deprivation therapy use for prostate cancer: role of the urologist.
Shahinian et al.
JNCI J Natl Cancer Inst 2006;98:839-845.
ABSTRACT | FULL TEXT  

Horses or Unicorns: Can Paying for Performance Make Quality Competition Routine?
Sage and Kalyan
Journal of Health Politics, Policy and Law 2006;31:531-556.
ABSTRACT  

Clinical Decision-Making in Blood Pressure Management of Patients with Diabetes Mellitus: An Oklahoma Physicians Resource/Research Network (OKPRN) Study.
Cotton et al.
J Am Board Fam Med 2006;19:232-239.
ABSTRACT | FULL TEXT  

Therapy Modifications in Response to Poorly Controlled Hypertension, Dyslipidemia, and Diabetes Mellitus
Rodondi et al.
ANN INTERN MED 2006;144:475-484.
ABSTRACT | FULL TEXT  

Comparison of Weighted Performance Measurement and Dichotomous Thresholds for Glycemic Control in the Veterans Health Administration
Pogach et al.
Diabetes Care 2006;29:241-246.
ABSTRACT | FULL TEXT  

An Endocrinologist-Supported Intervention Aimed at Providers Improves Diabetes Management in a Primary Care Site: Improving Primary Care of African Americans with Diabetes (IPCAAD) 7
Phillips et al.
Diabetes Care 2005;28:2352-2360.
ABSTRACT | FULL TEXT  

A Process for Measuring the Quality of Cancer Care: The Quality Oncology Practice Initiative
Neuss et al.
JCO 2005;23:6233-6239.
ABSTRACT | FULL TEXT  

Physicians And Ambulatory Electronic Health Records
Bates
Health Aff (Millwood) 2005;24:1180-1189.
ABSTRACT | FULL TEXT  

Clinical Practice Guidelines and Quality of Care for Older Patients With Multiple Comorbid Diseases: Implications for Pay for Performance
Boyd et al.
JAMA 2005;294:716-724.
ABSTRACT | FULL TEXT  

Impact of an Electronic Medical Record on Diabetes Quality of Care
O'Connor et al.
Ann Fam Med 2005;3:300-306.
ABSTRACT | FULL TEXT  

Are Patients' Ratings of Their Physicians Related to Health Outcomes?
Franks et al.
Ann Fam Med 2005;3:229-234.
ABSTRACT | FULL TEXT  

Report Cards and Nursing Homes
Castle and Lowe
Gerontologist 2005;45:48-67.
ABSTRACT | FULL TEXT  

Is Risk-Adjustor Selection More Important Than Statistical Approach for Provider Profiling? Asthma as an Example
Huang et al.
Med Decis Making 2005;25:20-34.
ABSTRACT  

The Vermont Diabetes Information System (VDIS): study design and subject recruitment for a cluster randomized trial of a decision support system in a regional sample of primary care practices
MacLean et al.
Clin Trials 2004;1:532-544.
ABSTRACT  

Payment for Quality in End-Stage Renal Disease
Himmelfarb et al.
J. Am. Soc. Nephrol. 2004;15:3263-3269.
FULL TEXT  

Professionalism Reconsidered: Physician Payment In A Small-Practice Environment
Cunningham
Health Aff (Millwood) 2004;23:36-47.
ABSTRACT | FULL TEXT  

The Role of Physician Specialty Board Certification Status in the Quality Movement
Brennan et al.
JAMA 2004;292:1038-1043.
ABSTRACT | FULL TEXT  

Variations In The Quality Of Care For Very-Low-Birthweight Infants: Implications For Policy
Rogowski et al.
Health Aff (Millwood) 2004;23:88-97.
ABSTRACT | FULL TEXT  

Opportunities and Challenges for Measuring Cost, Quality, and Clinical Effectiveness in Health Care
Fishman et al.
Med Care Res Rev 2004;61:124S-143S.
ABSTRACT  

Physicians and Corporations: A Corporate Transformation of American Medicine?
Casalino
Journal of Health Politics, Policy and Law 2004;29:869-884.
 

Which data source in clinical performance assessment? A pilot study comparing self-recording with patient records and observation
Spies et al.
Int J Qual Health Care 2004;16:65-72.
ABSTRACT | FULL TEXT  

How to make a silk purse from a sow's ear--a comprehensive review of strategies to optimise data for corrupt managers and incompetent clinicians
Pitches et al.
BMJ 2003;327:1436-1439.
FULL TEXT  

Benefits of and Barriers to Large Medical Group Practice in the United States
Casalino et al.
Arch Intern Med 2003;163:1958-1964.
ABSTRACT | FULL TEXT  

Physician Clinical Performance Assessment: Prospects and Barriers
Landon et al.
JAMA 2003;290:1183-1189.
ABSTRACT | FULL TEXT  

Improving the Quality of Care -- Can We Practice What We Preach?
Steinberg
NEJM 2003;348:2681-2683.
FULL TEXT  

Accepting Critically Ill Transfer Patients: Adverse Effect on a Referral Center's Outcome and Benchmark Measures
Rosenberg et al.
ANN INTERN MED 2003;138:882-890.
ABSTRACT | FULL TEXT  

Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls
Powell et al.
Qual Saf Health Care 2003;12:122-128.
ABSTRACT | FULL TEXT  

The Quality of Quality Measurement in U.S. Nursing Homes
Mor et al.
Gerontologist 2003;43:37-46.
ABSTRACT | FULL TEXT  

A Research Agenda For Bridging The 'Quality Chasm'
Fernandopulle et al.
Health Aff (Millwood) 2003;22:178-190.
ABSTRACT | FULL TEXT  

Chronic Illness Management: What Is the Role of Primary Care?
Rothman and Wagner
ANN INTERN MED 2003;138:256-261.
ABSTRACT | FULL TEXT  

Risk Stratification and Comorbidity
Ferraris and Ferraris
Card Surg Adult 2003;2:187-224.
FULL TEXT  

From Physician to Consumer: The Effectiveness of Strategies to Manage Health Care Utilization
Flynn et al.
Med Care Res Rev 2002;59:455-481.
ABSTRACT  

Large Employers' New Strategies in Health Care
Galvin and Milstein
NEJM 2002;347:939-942.
FULL TEXT  

Can We Be Certain of PRO Accuracy and Accountability? Inaccurate "Report Cards" Generated by Medicare Claims Data as a Marker for Laboratory Testing in Diabetic Patients
Gruber and Rudnitsky
American Journal of Medical Quality 2002;17:171-174.
ABSTRACT  

Why Is There A Quality Chasm?
Newhouse
Health Aff (Millwood) 2002;21:13-25.
ABSTRACT | FULL TEXT  

Attachment Theory: A Model for Health Care Utilization and Somatization
Ciechanowski et al.
Psychosom. Med. 2002;64:660-667.
ABSTRACT | FULL TEXT  

Health Care Report Cards: Implications for Vulnerable Patient Groups and the Organizations Providing Them Care
Davies et al.
Journal of Health Politics, Policy and Law 2002;27:379-400.
ABSTRACT  

Performance Profiles: The Influence of Patient Satisfaction Data on Physicians' Practice
Rider and Perrin
Pediatrics 2002;109:752-757.
ABSTRACT | FULL TEXT  

Profiling Care Provided by Different Groups of Physicians: Effects of Patient Case-Mix (Bias) and Physician-Level Clustering on Quality Assessment Results
Greenfield et al.
ANN INTERN MED 2002;136:111-121.
ABSTRACT | FULL TEXT  

Cardiac surgery report cards: comprehensive review and statistical critique
Shahian et al.
Ann. Thorac. Surg. 2001;72:2155-2168.
ABSTRACT | FULL TEXT  

Incorporating the Results of Diabetes Research Into Clinical Practice: Celebrating 25 years of Diabetes Research and Training Center translation research
Clark et al.
Diabetes Care 2001;24:2134-2142.
FULL TEXT  

Improving the Quality of Medical Care: Building Bridges Among Professional Pride, Payer Profit, and Patient Satisfaction
Grol
JAMA 2001;286:2578-2585.
ABSTRACT | FULL TEXT  

The impact of statistical adjustment on economic profiles of interventional cardiologists
Cowper et al.
J Am Coll Cardiol 2001;38:1416-1423.
ABSTRACT | FULL TEXT  

Relationship Between Asthma Medication and Antibiotic Use
Glauber et al.
Chest 2001;120:1485-1492.
ABSTRACT | FULL TEXT  

California Provider Group Report Cards: What Do They Tell Us?
Simon and Monroe
American Journal of Medical Quality 2001;16:61-70.
ABSTRACT  

Transforming Insurance Coverage Into Quality Health Care: Voltage Drops From Potential to Delivered Quality
Eisenberg and Power
JAMA 2000;284:2100-2107.
ABSTRACT | FULL TEXT  

Performance Measurement Through Audit, Feedback, and Profiling as Tools for Improving Clinical Care
Weiss and Wagner
Chest 2000;118:53S-58S.
ABSTRACT | FULL TEXT  

Inequality in Quality: Addressing Socioeconomic, Racial, and Ethnic Disparities in Health Care
Fiscella et al.
JAMA 2000;283:2579-2584.
ABSTRACT | FULL TEXT  

Unreliability of Physician "Report Cards" to Assess Cost and Quality of Care
Huff et al.
JAMA 2000;283:51-54.
FULL TEXT  

Recertification in the United States
Norcini
BMJ 1999;319:1183-1185.
FULL TEXT  

Governmental Databases, Hospital Information Systems, and Clinical Outcomes: Big Brother or Big Help?
Fleisher and Barash
Anesth. Analg. 1999;89:811-811.
FULL TEXT