This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (79)  • Contact me when this article is cited  Related Content  • Related letter  • Similar articles in JAMA  Topic Collections  • Cardiovascular System  • Quality of Care  • Quality of Care, Other  • Surgery  • Surgical Interventions  • Cardiovascular/ Cardiothoracic Surgery  • Gastrointestinal/ Upper Foregut  • Orthopedic Surgery  • Thoracic Surgery  • Alert me on articles by topic  Social Bookmarking   What's this?

The Problem With Small Sample Size

Justin B. Dimick, MD; H. Gilbert Welch, MD, MPH; John D. Birkmeyer, MD

JAMA. 2004;292:847-851.

Context  Surgical mortality rates are increasingly used to measure hospital quality. It is not clear, however, how many hospitals have sufficient caseloads to reliably identify quality problems.

Objective  To determine whether the 7 operations for which mortality has been advocated as a quality indicator by the Agency for Healthcare Research and Quality (coronary artery bypass graft [CABG] surgery, repair of abdominal aortic aneurysm, pancreatic resection, esophageal resection, pediatric heart surgery, craniotomy, hip replacement) are performed frequently enough to reliably identify hospitals with increased mortality rates.

Design and Setting  The US national average mortality rates and hospital caseloads of the 7 operations were determined using the 2000 Nationwide Inpatient Sample (NIS), and sample size calculations were performed to determine the minimum caseload necessary to reliably detect increased mortality rates in poorly performing hospitals. A 3-year hospital caseload was used for the baseline analysis, and poor performance was defined as a mortality rate double the national average.

Main Outcome Measure  Proportion of hospitals in the United States that performed more than the minimum caseload for each operation.

Results  The national average mortality rates for the 7 procedures examined ranged from 0.3% for hip replacement to 10.7% for craniotomy. Minimum hospital caseloads necessary to detect a doubling of the mortality rate were 64 cases for craniotomy, 77 for esophageal resection, 86 for pancreatic resection, 138 for pediatric heart surgery, 195 for repair of abdominal aortic aneurysm, 219 for CABG surgery, and 2668 for hip replacement. For only 1 operation did the majority of hospitals exceed the minimum caseload, with 90% of hospitals performing CABG surgery having a caseload of 219 or higher. For the remaining operations, only a small proportion of hospitals met the minimum caseload: craniotomy (33%), pediatric heart surgery (25%), repair of abdominal aortic aneurysm (8%), pancreatic resection (2%), esophageal resection (1%), and hip replacement (<1%).

Conclusion  Except for CABG surgery, the operations for which surgical mortality has been advocated as a quality indicator are not performed frequently enough to judge hospital quality.

Author Affiliations: VA Outcomes Group, Department of Veterans Affairs Medical Center, White River Junction, Vt (Drs Dimick and Welch); Center for the Evaluative Clinical Sciences, Dartmouth Medical School, Hanover, NH (Drs Dimick and Welch); and Michigan Surgical Collaborative for Outcomes Research and Evaluation (M-SCORE), Department of Surgery, University of Michigan Medical Center, Ann Arbor (Drs Dimick and Birkmeyer).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

RELATED LETTER

Surgical Mortality, Hospital Quality, and Small Sample Size
Mayer Brezis and Avraham Oren
JAMA. 2005;293(5):553.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Public reporting and pay-for-performance programs in perioperative medicine: Are they meeting their goals?
LINDENAUER
Cleveland Clinic Journal of Medicine 2009;76:S3-S8.
ABSTRACT | FULL TEXT  

Diabetes Surgery: A New Approach to an Old Disease
Rubino et al.
Diabetes Care 2009;32:S368-S372.
FULL TEXT  

The Prevalence of Perioperative Visual Loss in the United States: A 10-Year Study from 1996 to 2005 of Spinal, Orthopedic, Cardiac, and General Surgery
Shen et al.
Anesth. Analg. 2009;109:1534-1545.
ABSTRACT | FULL TEXT  

Toward quality improvement of thoracic aortic surgery: estimating volume-outcome effect from nationwide survey
Miyata et al.
Eur. J. Cardiothorac. Surg. 2009;36:517-521.
ABSTRACT | FULL TEXT  

Patient and Hospital Characteristics on the Variance of Perioperative Outcomes for Pancreatic Resection in the United States: A Plea for Outcome-Based and Not Volume-Based Referral Guidelines
Teh et al.
Arch Surg 2009;144:713-721.
ABSTRACT | FULL TEXT  

Composite Measures For Predicting Surgical Mortality In The Hospital
Dimick et al.
Health Aff (Millwood) 2009;28:1189-1198.
ABSTRACT | FULL TEXT  

Safety and efficacy of video-assisted versus conventional lung resection for lung cancer.
Farjah et al.
J. Thorac. Cardiovasc. Surg. 2009;137:1415-1421.
ABSTRACT | FULL TEXT  

The European Thoracic Database project: composite performance score to measure quality of care after major lung resection
Brunelli et al.
Eur. J. Cardiothorac. Surg. 2009;35:769-774.
ABSTRACT | FULL TEXT  

From Tasks To Processes: The Case For Changing Health Information Technology To Improve Health Care
Walker and Carayon
Health Aff (Millwood) 2009;28:467-477.
ABSTRACT | FULL TEXT  

Invited commentary.
Jacobs et al.
Ann. Thorac. Surg. 2009;87:587-588.
FULL TEXT  

Measuring the Quality of Surgical Outcomes in General Thoracic Surgery: Should Surgical Volume Be Used to Direct Patient Referrals?
Kozower et al.
Ann. Thorac. Surg. 2008;86:1405-1408.
FULL TEXT  

Evaluating Quality in Small-Volume Hospitals
Drye and Chen
Arch Intern Med 2008;168:1249-1251.
FULL TEXT  

Impact of Case Volume on Hospital Performance Assessment
O'Brien et al.
Arch Intern Med 2008;168:1277-1284.
ABSTRACT | FULL TEXT  

Low-volume coronary artery bypass surgery: Measuring and optimizing performance.
Shahian and Normand
J. Thorac. Cardiovasc. Surg. 2008;135:1202-1209.
FULL TEXT  

Effect of procedural volume on outcome of coronary artery bypass graft surgery in Japan: Implication toward public reporting and minimal volume standards.
Miyata et al.
J. Thorac. Cardiovasc. Surg. 2008;135:1306-1312.
ABSTRACT | FULL TEXT  

Measuring and Improving the Quality of Care for Abdominal Aortic Aneurysm Surgery
Dimick and Upchurch
Circulation 2008;117:2534-2541.
FULL TEXT  

Achieving Excellence
Bove
Ann. Thorac. Surg. 2008;85:368-370.
FULL TEXT  

Real-World Replication of Randomized Controlled Trial Results for Carotid Endarterectomy
Feasby et al.
Arch Neurol 2007;64:1496-1500.
ABSTRACT | FULL TEXT  

Bariatric Surgery for Morbid Obesity
DeMaria
NEJM 2007;356:2176-2183.
FULL TEXT  

Quality Assessment in High-Acuity Surgery: Volume and Mortality Are Not Enough
Vollmer et al.
Arch Surg 2007;142:371-380.
ABSTRACT | FULL TEXT  

Quality Measurement in Adult Cardiac Surgery: Part 1--Conceptual Framework and Measure Selection
Shahian et al.
Ann. Thorac. Surg. 2007;83:S3-S12.
FULL TEXT  

Elective surgery for aortic abdominal aneurysm: comparison of English outcomes with those elsewhere
Filipovic et al.
J. Epidemiol. Community Health 2007;61:226-231.
ABSTRACT | FULL TEXT  

Assessing the Accuracy of Hospital Clinical Performance Measures
Normand et al.
Med Decis Making 2007;27:9-20.
ABSTRACT  

Mortality in Patients Hospitalized for Asthma Exacerbations in the United States
Krishnan et al.
Am. J. Respir. Crit. Care Med. 2006;174:633-638.
ABSTRACT | FULL TEXT  

Massachusetts Cardiac Surgery Report Card: Implications of Statistical Methodology
Shahian et al.
Ann. Thorac. Surg. 2005;80:2106-2113.
ABSTRACT | FULL TEXT  

Should Coronary Artery Bypass Grafting Be Regionalized?
Nallamothu et al.
Ann. Thorac. Surg. 2005;80:1572-1581.
FULL TEXT  

Monitoring surgical mortality
Baxter
BMJ 2005;330:1098-1099.
FULL TEXT  

Surgical Mortality, Hospital Quality, and Small Sample Size
Brezis and Oren
JAMA 2005;293:553-553.
FULL TEXT