 |
|
Effectiveness of Teaching Quality Improvement to CliniciansA Systematic Review
Romsai T. Boonyasai, MD, MPH;
Donna M. Windish, MD, MPH;
Chayan Chakraborti, MD;
Leonard S. Feldman, MD;
Haya R. Rubin, MD, PhD;
Eric B. Bass, MD, MPH
JAMA. 2007;298:1023-1037.
ABSTRACT
| |
Context Accreditation requirements mandate teaching quality improvement (QI) concepts to medical trainees, yet little is known about the effectiveness of teaching QI.
Objectives To perform a systematic review of the effectiveness of published QI curricula for clinicians and to determine whether teaching methods influence the effectiveness of such curricula.
Data Sources The electronic literature databases of MEDLINE, EMBASE, CINAHL, and ERIC were searched for English-language articles published between January 1, 1980, and April 30, 2007. Experts in the field of QI were queried about relevant studies.
Study Selection Two independent reviewers selected studies for inclusion if the curriculum taught QI principles to clinicians and the evaluation used a comparative study design.
Data Extraction Information about the features of each curriculum, its use of 9 principles of adult learning, and the type of educational and clinical outcomes were extracted. The relationship between the outcomes and the number of educational principles used was assessed.
Results Of 39 studies that met eligibility criteria, 31 described team-based projects; 37 combined didactic instruction with experiential learning. The median number of adult learning principles used was 7 (range, 2-8). Evaluations included 22 controlled trials (8 randomized and 14 nonrandomized) and 17 pre/post or time series studies. Fourteen studies described educational outcomes (attitudes, knowledge, or skills or behaviors) and 28 studies described clinical process or patient outcomes. Nine of the 10 studies that evaluated knowledge reported only positive effects but only 2 of these described a validated assessment tool. The 6 assessments of attitudes found mixed results. Four of the 6 studies on skill or behavior outcomes reported only positive effects. Eight of the 28 studies of clinical outcomes reported only beneficial effects. Controlled studies were more likely than other studies to report mixed or null effects. Only 4 studies evaluated both educational and clinical outcomes, providing limited evidence that educational outcomes influence the clinical effectiveness of the interventions.
Conclusions Most published QI curricula apply sound adult learning principles and demonstrate improvement in learners' knowledge or confidence to perform QI. Additional studies are needed to determine whether educational methods have meaningful clinical benefits.
INTRODUCTION
Thought leaders and professional organizations in medicine and nursing have endorsed teaching quality improvement (QI) to clinicians.1-5 The Accreditation Council on Graduate Medical Education has linked the accreditation of postgraduate training programs with demonstration that their physicians-in-training achieve competency in systems-based practice and practice-based learning and improvement.6 However, the potential effect of teaching QI methods to clinicians is uncertain. Most reviews of QI interventions have described educational interventions meant to improve clinicians' knowledge of or adherence to guidelines instead of providing them with skills to implement systems changes themselves.7-9 One of 2 reviews that examined teaching QI to clinicians focused on establishing a framework for organizing QI content for learners10 and the other was not a systematic review.5
We therefore systematically reviewed the literature to determine the effect of teaching QI to clinicians, and to determine whether the use of adult learning principles or the presence of specific curricular features is associated with the effectiveness of these educational interventions.
METHODS
Literature Search
We searched for relevant studies published through April 2007 using electronic literature databases, including MEDLINE, Excerpta Medica (EMBASE), the Cumulative Index of Nursing and Allied Health Literature (CINAHL), and the Education Resource Information Center (ERIC).
The search strategy was limited to English-language articles published between January 1, 1980, and April 30, 2007, using the Medical Subject Headings (MeSH) health care economics and organizations; health care quality, access, and evaluation; health care facilities; manpower and services; health services administration; information management; informatics; medical informatics; and systems analysis. These were combined with the quality-related terms of quality improvement, quality management, continuous improvement, performance improvement, (improve, improving, manage, or managing and quality), QI, CQI, TQM, quality assurance, quality assessment, patient safety, practice based, system based, systems based, plan do study act, plan do check act, pdsa, pdca, six sigma, and lean management and the education-related terms of curriculum, curricul$, educat$, teach$, train$, learn$, and the MeSH term curriculum. The MeSH terms were exploded to include all associated subheadings, and the core search was limited using the terms NOT comment or editorial or letter.
We also queried experts in the field of QI and hand-searched the references of all included studies, relevant review articles, and the table of contents of key journals from April 2006 through May 2007. ProCite version 5.0.3 (Thompson ISI ResearchSoft, Berkeley, California) was used to store and track the search results.
Eligibility Criteria
Quality improvement theory was defined as a set of principles that involve knowledge, skills, and methods used to evaluate and implement change in a health care system using a systems-based approach.5 Curriculum was defined as a formal supervised program for changing knowledge, skills, or behaviors.11 Clinician was defined as any practitioner who provides direct medical care to patients, excluding dentists and clinical laboratory workers. Studies were included if they described both a curriculum that teaches QI theory to clinicians and a comparative evaluation (ie, pre/post evaluations, time series, or randomized or nonrandomized controlled evaluation). Studies were excluded if they did not (1) occur in North America, Western Europe, Australia, New Zealand, or Japan; (2) teach QI theory; (3) describe a curricular intervention; (4) target clinicians; (5) include original data; (6) have a full article available for review; or (7) publish the results of a comparative evaluation. Studies also were excluded if they were published before 1980 or were not written in English.
These criteria were chosen because industrial process improvement methods first became popular in the 1980s (no eligible studies were published before 1995) and because we were primarily interested in the effects of QI curricula in the US health care system and other similarly developed medical systems. We included curricula for practicing physicians as well as for medical trainees because QI concepts should be relevant to both groups.
Title and Abstract Review
Eligible studies were identified through 2 phases. In the first phase, 2 authors (D.M.W. and L.S.F. or R.T.B. and C.C.) independently reviewed the titles of citations to assess eligibility for review. Citations were returned for adjudication if disagreement occurred. If agreement could not be reached, the citation was retained for further review.
In the second phase, the abstracts of remaining citations were independently reviewed by 2 pairs of reviewers using forms for determining eligibility criteria. If reviewers disagreed, the study article was returned for adjudication until agreement was reached. If eligibility could not be determined because an abstract had insufficient information, the full article was retrieved. Reviewers were not masked to author or journal name because previous work has shown that such masking is unlikely to make a significant difference in the results of data abstraction.12
Study Review
All remaining studies were retrieved for data abstraction. Two reviewer pairs (R.T.B. and C.C. or D.M.W. and L.S.F.) used a standardized form to confirm eligibility of the article. Reviewers resolved disagreements by consensus. Eligible studies were abstracted using a serial review process. Three reviewers divided the primary reviews and one reviewer checked all initial abstractions (R.T.B., L.S.F., C.C., and D.M.W.). Extracted information assessed the curricular context, characteristics of targeted learners, educational strategies, evaluation methods, and results. Quality improvement curricular content was based on the Institute for Healthcare Improvement (IHI) domains of knowledge for improving health care.13 The 8 IHI domains are a well-recognized framework for classifying QI knowledge.10 They include customer knowledge; health care as a system; variation and measurement; leading, following, and making change; collaboration; developing new, locally useful knowledge; social context and accountability; and professional knowledge. Educational content that overlapped with more than 1 domain was classified into all relevant domains. Studies also were classified according to whether they involved other types of noneducational QI interventions such as audit, feedback, or patient registries by adapting the Cochrane Effective Practice and Organization of Care taxonomy.14
We hypothesized that an association exists between using sound adult learning principles and curriculum effectiveness. To evaluate the use of adult learning principles, a list of fundamental principles was developed based on a review of 5 major educational theories: andragogy, social cognitive theory, self-directed learning, constructivism, and reflective practice.15 Using this list, curricula were assessed for whether they fully or partially exhibited 9 characteristics: (1) enabling learners to be active participants, (2) providing content relating to learners' current experiences, (3) assessing learners' needs and tailoring teaching to their past experiences, (4) allowing learners to identify and pursue their own learning goals, (5) allowing learners to practice their learning, (6) supporting learners during self-directed learning, (7) providing feedback to learners, (8) facilitating learner self-reflection, and (9) role-modeling behaviors. The reviewers relied on the authors' descriptions to determine if a principle was used. For example, a curriculum was classified as fully exhibiting facilitation of reflective learning if the article specifically described using learner self-reflection, or partially exhibiting that principle if it described features likely to encourage self-reflection, such as asking learners to discuss lessons learned.
The reliability of all categorical questions was assessed for the first 27 articles abstracted. Overall agreement was 91.5% and interrater reliability was good ( = 0.77) for data abstraction. Interrater reliability for assessing adult learning principles was moderate ( = 0.51).
Data Synthesis and Statistical Analysis
Due to the heterogeneity of results, outcomes were classified as beneficial effects, no effects, or detrimental effects based on whether differences from either baseline (in uncontrolled studies) or controls were statistically significant at the P value level of .05 or less. If the authors did not calculate statistical significance, an effect was counted as beneficial or detrimental if the absolute change or difference in an outcome measure was greater than 10%. We used 2-tailed  2 statistics to assess differences in proportions. All statistical analyses were performed using Intercooled Stata version 9.2 (StataCorp, College Station, Texas).
RESULTS
Characteristics of Eligible Studies
Of 16 278 citations identified, 39 studies met eligibility criteria (Figure). Four study articles described evaluations of 2 separate curricula.16-19 Table 1 summarizes the characteristics of the studies. Eighty percent of studies were published in the last 5 years. All targeted physicians, medical trainees, nurses, or nursing students. Many curricula also taught other clinicians. Ten studies targeted trainees20-29 and 29 targeted nontrainees.16-19,30-54 Two of the IHI knowledge domains were covered by all of the curricula: leading, following, and making change; and developing new, locally useful knowledge. Nearly all of the studies described the use of didactic instruction in combination with experiential learning.16-29,31-33,35-54 Only 8 of the studies used a randomized controlled group design.16-17,31-32,35-36,40-41 Table 2 and Table 3 describe the characteristics of individual curricula, which were grouped into 3 categories to facilitate analysis: curricula for trainees, curricula for nontrainees that have an educational focus, and curricula for nontrainees that are included in a multifaceted QI intervention.
|
|
|
|
Figure. Literature Search and Study Selection Process
|
|
|
|
|
|
|
Table 1. Characteristics of Quality Improvement (QI) Curriculum Studies (N = 39)a
|
|
|
|
|
|
|
Table 2. Characteristics of Quality Improvement (QI) Curricula With Controlled Evaluations
|
|
|
|
|
|
|
Table 3. Characteristics of Quality Improvement (QI) Curricula With Uncontrolled Evaluations
|
|
|
Features of Curricula for Trainees
Of the 10 curricula targeting trainees, 3 involved either medical or nursing students.26-27,29 Program details varied greatly: one involved preclinical nursing students observing patients' perspectives of hospital wards for 10 weeks before learning and discussing how to apply QI methods,27 while another was a 4-year longitudinal program that integrated didactic instruction and QI activities into an existing medical school curriculum.26
Seven studies involving residents had more similarities.20-25,28 All took place during ambulatory care assignments or electives, all combined didactic instruction with participation in QI activities, and none included noneducational QI interventions. Five curricula were integrated into a 4-week rotation,20-22,24-26 while 2 held weekly or biweekly meetings for a year.23, 28
Features of Curricula for Nontrainees That Have an Educational Focus
Five of the 29 studies (17%) involving nontrainees described their primary focus as conveying QI concepts to learners.30-34 Peters et al33 described a 4-month medical school faculty development program teaching QI concepts. Irvine Doran et al31 taught QI concepts to hospital-based clinicians and encouraged application of these skills in their hospitals. The remaining 3 studies described brief (1- to 7-day) continuing education–style seminars.30, 32, 34
Features of Curricula for Nontrainees That Are Included in a Multifaceted QI Intervention
Twenty-four of the 29 studies (83%) involving nontrainees were QI interventions with educational components.16-19,35-54 All obtained administrative support for the intervention and provided faculty with QI expertise. We have grouped these studies into 4 subcategories according to characteristics of the overall QI intervention.
QI Training Session Supplementing Noneducational QI Interventions. Five studies described QI curricula integrated into larger QI interventions primarily managed by project organizers.17, 35-38 All taught QI concepts and methods during an initial training session associated with audit and feedback. Learners then implemented QI interventions with personalized faculty support. Most of these interventions also provided learners with a package of predeveloped QI instruments, such as patient educational materials or customized flow sheets.17, 35-37
QI Curricula in IMPROVE and IDEAL Collaboratives. Nineteen of the eligible studies (49%) described teaching learners through QI collaboratives, whereby learners attended facilitated workshops in which they shared ideas and taught each other QI concepts related to a clinical process or disease focus. The Improving Prevention Through Organization, Vision and Empowerment (IMPROVE) and Improving Diabetes Care Through Empowerment, Active Collaboration and Leadership (IDEAL) collaboratives taught QI concepts to primary care teams focused on preventive care and diabetes management.40-41 These curricula offered didactic instruction and interactive discussions during 7 half-day workshops over an 18- to 22-month period, interspersed with QI activities that were supported by faculty telephone calls and site visits. Instead of providing learners with a preexisting package of QI interventions, they focused on facilitating networking and sharing of ideas by learners, who implemented the ideas they believed to be most appropriate for their sites.
QI Curricula in Breakthrough Series Collaboratives. Twelve studies described curricula based on the IHI-sponsored Breakthrough Series.16, 18-19,42-49,54 Breakthrough Series collaboratives teach a QI approach using the chronic care model (a multidisciplinary approach to providing decision support and supporting patient self-management) and the model for improvement (a conceptual model for implementing and testing process change).55 They require use of an electronic patient registry and have been used with more than 20 different conditions throughout the world. Similar to the IMPROVE and IDEAL programs, learners in Breakthrough Series curricula also attend multiple QI workshops over a 6- to 18-month period, interspersed with QI activities. However, Breakthrough Series collaboratives emphasize incremental small tests of change. For example, learners in one Breakthrough Series program implemented and tested an average of 42 small process changes over 12 months,42 in contrast to learners in an IDEAL collaborative who implemented and tested 1 comprehensive process change in 18 months.40
QI Curricula in Interventions Using Other Collaborative Models. Five studies described QI curricula in the context of other collaboratives. Three were described as modifications of the Breakthrough Series model.50-52 One used a unique conceptual model for teaching QI in the neonatal intensive care unit.53 One supplemented QI training for coronary bypass graft surgery teams with observational site visits among learners' hospitals.39 All 5 studies presented learners with audit and feedback at an initial learning session followed by teleconferences or site visits to facilitate collaboration during QI projects.
Use of Adult Learning Principles
The extent to which QI curricula used adult learning principles is presented in eTable 1. Most studies described more than 6 of the 9 adult learning principles. The studies involving trainees often provided richer descriptions of the educational intervention, and as a group they described the presence of more adult learning principles than those involving nontrainees (median number of principles 8 vs 6). Among the 10 studies involving trainees,20-29 all curricula allowed learners to actively contribute to their education, engaged learners with topics that were relevant to their current experiences, helped learners direct their own learning, facilitated practice through QI activities, and described feedback strategies. Only 3 studies (30%) described role modeling as a teaching method.23, 26, 29
The 29 studies of nontrainees described using adult learning principles less often than studies of trainees but more often than has been described for traditional continuing education.56 All of these studies facilitated learners' active contributions and addressed topics relevant to their current experiences.16-19,30-54 However, only 1 of these studies (3%) used role modeling,33 only 6 studies (21%) described a needs assessment,17-18,33, 35, 42, 46 and only 18 studies (62%) reported giving feedback to learners.16-18,31, 35-39,43, 46-50,52-54
Results of Evaluations
Because standard measures for assessing QI curricular effectiveness do not exist,57 the studies in this review present heterogeneous outcomes (details are provided in eTable 2). Table 4 and Table 5 summarize the evaluation methods and direction of outcomes, organized by type of learner, type of intervention, and study design. Only 3 of the studies (8%) described both clinical and educational outcomes.22, 29, 48 Additionally, one study19 described behavioral changes related to clinical outcomes reported in another study.18
|
|
|
|
Table 4. Evaluation Methods and Direction of Educational Outcomes
|
|
|
|
|
|
|
Table 5. Evaluation Methods and Direction of Clinical Outcomes
|
|
|
With the exception of a single detrimental effect in learner attitudes,29 all outcomes had either positive or null effects. Twelve studies (31%) reported only beneficial effects,17, 22-23,25, 27-28,30, 33-34,39, 43, 46 24 studies (62%) reported mixed effects (positive, null, and detrimental),19-21,24, 26, 29, 31-32,35-38,41-42,44-45,47-54 and 3 studies found only null effects16, 18, 40 (Table 6). Studies most frequently reported improved knowledge and rarely found benefits in patient outcomes. Fewer controlled studies reported only positive outcomes compared with those using uncontrolled designs (4 of 22 controlled studies vs 8 of 17 uncontrolled studies; P = .05).
|
|
|
|
Table 6. Summary of Evaluation Results of Quality Improvement Curricula
|
|
|
Educational Outcomes
Of 14 studies describing educational outcomes,19, 21-24,26-27,29-34,48 4 described curricula for residents21-24; 3 for medical or nursing students26-27,29; 1 for medical school faculty33; and 6 for nontrainee clinicians.19, 30-32,34, 48 The direction of outcomes did not appear to differ by the training level of the learners.
Of the 10 studies (71%) that reported the effect of teaching QI on knowledge,21-22,24, 29-34,48 all 5 involving trainees reported improvement in either perceived knowledge24, 27 or scores on a QI knowledge test.21-22,24, 29 Four of 5 studies involving nontrainees found improvement in either perceived30, 33 or tested32, 34 knowledge. One study of hospital-based clinicians found no change in scores on a QI knowledge examination.31 Only 2 studies used the same knowledge test (Quality Improvement Knowledge Assessment Test). 21-22
Five of 6 studies describing attitudinal changes involved trainees.21, 23-24,26, 29 One found residents' confidence in their ability to provide good diabetes care improved23; another reported no change in residents' interest in QI.24 One curriculum reported improved medical students' attitudes toward managed care,26 while 2 studies found mixed attitudes toward QI interventions.21, 29 In these studies, learners reported improved confidence in their own or in their organization's ability to engage in QI projects but no change was seen in their confidence to sustain systemic change. Only 1 study described attitudinal changes in nontrainees. That study found that learners' confidence in their organization's system for diabetes management improved but their confidence in their organization's ability to support staff or community linkages did not change.48
Six curricula evaluated changes in learners' skills or behaviors.19, 24, 30-33 The only program involving trainees found that residents' perceived ability to implement QI projects improved.24 Of the 5 studies involving nontrainees, one curriculum for hospital-based teams reported improvement in functional group interactions using a validated scale,31 another curriculum for medical school faculty described improvement in their perceived ability to teach QI,33 and 3 studies described improvement in self-reported ability to implement QI projects and greater involvement in QI activities.24, 30, 32 A final study found no changes in organizational behaviors for 4 of 6 chronic care model domains.19
Clinical Outcomes
The 5 curricula involving trainees that evaluated clinical outcomes generally reported positive results.20, 22, 25, 28-29 Most used uncontrolled evaluation methods with small sample sizes. Varkey et al22 reported 18% to 37% improvement in documentation processes before and after the QI interventions but did not assess statistical significance. Coleman et al25 reported 24% to 34% improvement in documentation processes (P = .001) and 26% improvement in diabetic microalbuminuria screening (P = .02). Mohr et al28 found significantly increased on-time chart availability (40%; P < .01), chart completeness (61%; P < .01), and immunization rates of infants (26%; P = .04). In a nonrandomized controlled study, Holmboe et al20 found that diabetic patients of residents in the intervention group had 12% more annual monofilament examinations (P = .02), 10% more baseline electrocardiograms (P = .01), and greater decreases in levels of hemoglobin A1c (1.1%; P = .001) and low-density lipoprotein cholesterol (20 mg/dL [0.52 mmol/L]; P = .01) than those followed up by control residents. No improvement, however, was seen in obtaining screening laboratory test results, providing immunizations, or patients' blood pressure.20 Gould et al29 described the only trainee-centered curriculum that integrated noneducational QI interventions, including chart audit tools, and patient education materials. The documentation of diabetic eye and foot examinations and patients' hemoglobin A1c levels improved significantly, although rates of documenting levels of hemoglobin A1c (which was high at baseline) did not change.29
All curricula that involved nontrainees and reported clinical outcomes were associated with other noneducational QI interventions. They reported a diverse set of outcomes, individually reporting a mean of 9.7 (range, 1-25) different clinical outcomes. The results of these studies were mixed and are detailed in eTable 2. The key outcomes of controlled studies are described herein.
Most studies that supplemented traditional audit and feedback with individual coaching in QI methods reported positive process outcomes and highlighted the importance of in-person faculty coaching.17, 35-38 One curriculum—described in 2 separate studies17, 35—randomized pediatric practices to receive audit and feedback with either cursory instruction in process improvement methods or extensive coaching in QI methods. Margolis et al17 found higher adjusted odds of providing 4 preventive services in interventional practices than in control practices (odds ratio, 4.6; 95% confidence interval [CI], 1.6-13.2). Rosenthal et al35 reported that rates of all anticipatory counseling measures increased 18.7% to 22.6% compared with controls (P = .002 to P = .001) but found no differences in parents' health knowledge or behaviors. Similar benefits were seen in 2 uncontrolled nursing home studies that used similar teaching methods.37-38 In contrast, McClellan et al36 found that adding didactic QI training and optional telephone-based faculty support to traditional audit and feedback was associated with improvement in only 4 of 10 measures of hemodialysis quality.
Evaluations of the IMPROVE and IDEAL collaborative models reported mostly null findings.40-41 Although Solberg et al41 found increased rates of pneumococcal immunizations and offers for cholesterol screening compared with controls, the authors found no significant differences in 10 other preventive care measures. Similarly, O’Connor et al40 reported no significant improvement compared with controls in 12 diabetes process measures.
By comparison, Breakthrough Series collaboratives16, 18, 42-49,54 reported more positive outcomes. Among the controlled studies, 1 reported only beneficial results for 5 measures of adherence to cardiac care guidelines,43 and 3 found significant improvements in 26 of 51 process outcomes and 4 of 12 patient outcomes related to asthma, cardiovascular care, and diabetes care.42, 44-45 However, not all Breakthrough Series curricula reported beneficial effects. Landon et al18 found no improvement in 6 process measures related to the human immunodeficiency virus (HIV) or in the proportion of controlled viral loads. Homer et al16 reported no change in use of asthma action plans or in the patients' symptoms and suggested that incomplete learner attendance may have influenced their findings. McInnes et al19 surveyed clinicians and administrators from the HIV collaborative described by Landon et al18 and theorized that the absence of organizational changes in 4 of 6 domains of the chronic care model may explain their null findings.
In contrast to QI collaboratives held in outpatient settings, those in highly controlled inpatient settings reported beneficial effects for a focused set of outcomes.39, 50-53 One curriculum involving neonatal intensive care units reported significantly higher adjusted odds of on-time surfactant delivery in the intervention group compared with controls who received only audit and feedback (odds ratio, 5.38; 95% CI, 2.84-10.20).53 Similarly, O’Connor et al39 reported a significantly lower standardized mortality ratio for cardiothoracic surgery at interventional hospitals compared with historical controls (standardized mortality ratio, 0.76; 95% CI, 0.67-0.90).
Association Between Use of Adult Learning Principles and Curriculum Outcomes
An exploratory analysis was performed to assess the relationship between use of adult learning principles and curriculum effectiveness by comparing the proportion of curricula reporting only beneficial effects with the number of adult learning principles described. Curricula with only beneficial effects, mixed effects, and only null effects all described a median of 7 principles (range 2-8). No relationships with this measure were seen using only controlled studies or those with only educational outcomes.
Analyses also were performed to assess if relationships exist between clinical outcomes and learner attendance, as well as between clinical and educational outcomes. Eight studies described incomplete attendance.16, 18, 25, 37, 41, 47-48,51 Among these, all 4 controlled evaluations found either no benefit or benefit in less than 25% of described outcomes.16, 18, 41, 51 Results for both educational and clinical evaluations were only available for 4 curricula. One reported improvements in both residents' QI knowledge and documentation processes,22 and another found improvements in practitioners' attitudes and in their patients' hemoglobin A1c levels.48 Neither study evaluated the statistical significance of their clinical outcomes. Gould et al29 reported increased knowledge and confidence in QI methods for 26 of 40 survey items and statistically significant improvement in diabetes-related processes and outcomes at medical students' preceptor sites. In contrast, the null clinical findings of an HIV collaborative were linked to an absence of change in organizational behaviors.18-19
Resources Used by Curricula
Only 6 studies described QI curricular costs.20-21,41, 45-46,48 In one program, residents spent an average of 119 hours in 4 weeks on their projects, faculty spent 6 hours per month teaching the curriculum, and project sponsors devoted 1 to 2 hours per week supporting residents' activities.21 These findings were comparable with other studies that reported time costs. Holmboe et al20 found that residents committed 4 half days and faculty committed 3 hours over 4 weeks, while Solberg et al41 reported that practicing clinicians spent 5 to 11.7 hours per month on their projects.
Monetary costs varied greatly and likely reflected resources needed to integrate QI curricula with other noneducational QI interventions. Ogrinc et al21 described the direct costs of their residency-based curriculum as negligible, whereas health care organizations each paid $12 500 plus travel expenses to enroll a 3-person team in a Breakthrough Series curriculum.45-46,48
COMMENT
To our knowledge, this is the first systematic review to evaluate the effect of teaching QI to clinicians. The 39 studies in our sample varied in their coverage of topics within the IHI domains of QI knowledge. All described teaching content required for implementing and evaluating local systems changes but few taught learners about the larger health care system or how to view the system from a patient's perspective.
Fourteen studies evaluating educational outcomes collectively suggest that using a variety of QI teaching strategies improves QI-related knowledge, as well as attitudes toward health systems and participation in QI activities for learners of many training levels and professions. However, relatively weak study designs and evaluation instruments hamper these assessments. Most educational evaluations use uncontrolled designs, which are vulnerable to secular trends and other biases. Furthermore, those that report changes in knowledge often rely on learners' self-report of knowledge, and the validity of this assessment method has been questioned.58 Of those using knowledge tests, only 2 studies use a validated test.21-22 The validity of learner attitudinal and behavioral assessments is similarly uncertain.
Curricula with clinical outcomes provide mixed evidence for improvement in processes and little evidence for beneficial patient outcomes. Those involving trainees report mostly beneficial outcomes in documentation and disease management processes but use relatively weak study designs.
All nontrainee curricula with clinical outcomes were components of multifaceted QI interventions that provided expert faculty and received administrative support. Of these, the IMPROVE and IDEAL programs placed greatest emphasis on theoretical QI content and included the fewest external interventions. Although their teaching strategies applied many adult learning principles, they nevertheless found little evidence for clinical benefit compared with nonparticipant controls. Their null findings stand in contrast to the more positive findings of programs that combined QI training with noncollaborative QI interventions. From these studies, it appears that providing learners with a package of QI tools and individualized coaching in QI methods can improve clinical outcomes compared with audit and feedback and limited QI training.
The positive outcomes associated with these programs and with the Breakthrough Series curricula also suggest that providing learners with access to preexisting performance data or process improvement tools can improve outcomes. For example, the IMPROVE and IDEAL curricula encouraged learners to collect their own performance data, whereas curricula in the Breakthrough Series collaboratives and those supplementing noncollaborative QI interventions provided learners with data from an electronic registry or external audits. The positive outcomes seen in curricula set in data-rich inpatient and intensive care environments also support the importance of access to performance data. Similarly, many curricula with positive outcomes also provided learners with preexisting decision support or patient education tools, which may have allowed them to focus on implementing interventions instead of developing their own tools. The relatively greater benefits seen in Breakthrough Series curricula also suggest that encouraging learners to implement and test interventions through several small cycles of trial and error may result in better outcomes than undertaking a single comprehensive intervention.
Nevertheless, the null findings in studies of HIV and pediatric Breakthrough Series collaboratives indicate that performing incremental tests of change and having ready access to performance data are insufficient to ensure beneficial clinical outcomes. The attendance issues discussed by Homer et al16 and the absence of organizational changes following the HIV collaborative18-19 may indicate that how well clinicians learned QI concepts influences clinical outcomes. The findings of 4 curricula for which educational and clinical outcomes are available18-19,22, 29, 48 suggest that clinical benefits occur when educational outcomes improve and are absent when learners’ behaviors do not change. Although our analysis did not find convincing evidence for an association between the use of adult learning principles and curriculum effectiveness, we were limited to the details that the authors published and we could not reliably assess important details such as attendance, learners' prior experiences, faculty QI expertise, or faculty teaching skill. Furthermore, counting curricula with only beneficial, mixed, or only null effects provides a crude measure of curriculum effectiveness.
This review has several limitations. First, it is subject to publication bias. Although we could not formally test for this, we did examine the sample size distribution for studies that had positive, mixed, and null effects, and found a relatively similar sample size distribution for each group, arguing against the presence of major publication bias. Second, our findings may not apply to medical education in all countries because we limited our review to studies that were published in English and occurred in countries with health care systems similar to the United States. In doing so, we inadvertently excluded countries such as Korea, Singapore, and Taiwan, which also have highly developed health care systems. However, in a retrospective review of the citations we excluded, we found no studies from those countries that would have met our eligibility criteria. Third, we focused on curricula that described results with comparisons to either baseline results or with controls, and thus excluded several descriptions of innovative QI curricula.59-63 We did so because we wished to examine the effects of teaching QI concepts instead of trying to describe the variety of ways in which they have been taught. Similarly, we excluded studies about QI interventions that did not provide at least an adequate description of the curriculum because we hoped to determine the effect of educational methods on curriculum effectiveness. Also, our analysis of the relationship between use of adult learning principles and curriculum effectiveness should be considered exploratory because no validated instrument exists for assessing the use of adult learning principles. Finally, our ability to combine outcomes was limited by their heterogeneity, which is an ongoing issue in systematic reviews of educational and organizational interventions in health care.64-65
Despite these limitations, we offer some recommendations for developing and evaluating future QI curricula. Published QI curricula teach methods for implementing systems change but infrequently address the IHI knowledge domains of health care as a system and customer knowledge. Because there was little evidence for improvement in patient knowledge or behaviors despite improvements in several measures of counseling processes, we recommend that future QI curricula also cover these IHI-defined domains. To advance the understanding of how teaching QI influences educational and clinical outcomes, educators should describe key features of their curricula, including learner and faculty characteristics, attendance, and learners' interventions. They also should compare learners with controls and present both educational and clinical outcomes. Part of the attractiveness of teaching QI to clinicians is that |
