This Article  • Abstract  • PDF  • eTables  • Author in the Room Teleconference  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (26)  • Contact me when this article is cited  Related Content  • Related article  • Similar articles in JAMA  Topic Collections  • Medical Practice  • Medical Education  • Quality of Care  • Quality of Care, Other  • Review  • Prognosis/ Outcomes  • Alert me on articles by topic  Social Bookmarking   What's this?

A Systematic Review

Romsai T. Boonyasai, MD, MPH; Donna M. Windish, MD, MPH; Chayan Chakraborti, MD; Leonard S. Feldman, MD; Haya R. Rubin, MD, PhD; Eric B. Bass, MD, MPH

JAMA. 2007;298(9):1023-1037.

ABSTRACT
Context  Accreditation requirements mandate teaching quality improvement (QI) concepts to medical trainees, yet little is known about the effectiveness of teaching QI.

Objectives  To perform a systematic review of the effectiveness of published QI curricula for clinicians and to determine whether teaching methods influence the effectiveness of such curricula.

Data Sources  The electronic literature databases of MEDLINE, EMBASE, CINAHL, and ERIC were searched for English-language articles published between January 1, 1980, and April 30, 2007. Experts in the field of QI were queried about relevant studies.

Study Selection  Two independent reviewers selected studies for inclusion if the curriculum taught QI principles to clinicians and the evaluation used a comparative study design.

Data Extraction  Information about the features of each curriculum, its use of 9 principles of adult learning, and the type of educational and clinical outcomes were extracted. The relationship between the outcomes and the number of educational principles used was assessed.

Results  Of 39 studies that met eligibility criteria, 31 described team-based projects; 37 combined didactic instruction with experiential learning. The median number of adult learning principles used was 7 (range, 2-8). Evaluations included 22 controlled trials (8 randomized and 14 nonrandomized) and 17 pre/post or time series studies. Fourteen studies described educational outcomes (attitudes, knowledge, or skills or behaviors) and 28 studies described clinical process or patient outcomes. Nine of the 10 studies that evaluated knowledge reported only positive effects but only 2 of these described a validated assessment tool. The 6 assessments of attitudes found mixed results. Four of the 6 studies on skill or behavior outcomes reported only positive effects. Eight of the 28 studies of clinical outcomes reported only beneficial effects. Controlled studies were more likely than other studies to report mixed or null effects. Only 4 studies evaluated both educational and clinical outcomes, providing limited evidence that educational outcomes influence the clinical effectiveness of the interventions.

Conclusions  Most published QI curricula apply sound adult learning principles and demonstrate improvement in learners' knowledge or confidence to perform QI. Additional studies are needed to determine whether educational methods have meaningful clinical benefits.

INTRODUCTION

Jump to Section  • Top  • Introduction  • Methods  • Results  • Comment  • Author information  • References

Thought leaders and professional organizations in medicine and nursing have endorsed teaching quality improvement (QI) to clinicians.^1-5 The Accreditation Council on Graduate Medical Education has linked the accreditation of postgraduate training programs with demonstration that their physicians-in-training achieve competency in systems-based practice and practice-based learning and improvement.⁶ However, the potential effect of teaching QI methods to clinicians is uncertain. Most reviews of QI interventions have described educational interventions meant to improve clinicians' knowledge of or adherence to guidelines instead of providing them with skills to implement systems changes themselves.^7-9 One of 2 reviews that examined teaching QI to clinicians focused on establishing a framework for organizing QI content for learners¹⁰ and the other was not a systematic review.⁵

We therefore systematically reviewed the literature to determine the effect of teaching QI to clinicians, and to determine whether the use of adult learning principles or the presence of specific curricular features is associated with the effectiveness of these educational interventions.

METHODS

Jump to Section  • Top  • Introduction  • Methods  • Results  • Comment  • Author information  • References

Literature Search

We searched for relevant studies published through April 2007 using electronic literature databases, including MEDLINE, Excerpta Medica (EMBASE), the Cumulative Index of Nursing and Allied Health Literature (CINAHL), and the Education Resource Information Center (ERIC).

The search strategy was limited to English-language articles published between January 1, 1980, and April 30, 2007, using the Medical Subject Headings (MeSH) health care economics and organizations; health care quality, access, and evaluation; health care facilities; manpower and services; health services administration; information management; informatics; medical informatics; and systems analysis. These were combined with the quality-related terms of quality improvement, quality management, continuous improvement, performance improvement, (improve, improving, manage, or managing and quality), QI, CQI, TQM, quality assurance, quality assessment, patient safety, practice based, system based, systems based, plan do study act, plan do check act, pdsa, pdca, six sigma, and lean management and the education-related terms of curriculum, curricul$, educat$, teach$, train$, learn$, and the MeSH term curriculum. The MeSH terms were exploded to include all associated subheadings, and the core search was limited using the terms NOT comment or editorial or letter.

We also queried experts in the field of QI and hand-searched the references of all included studies, relevant review articles, and the table of contents of key journals from April 2006 through May 2007. ProCite version 5.0.3 (Thompson ISI ResearchSoft, Berkeley, California) was used to store and track the search results.

Eligibility Criteria

Quality improvement theory was defined as a set of principles that involve knowledge, skills, and methods used to evaluate and implement change in a health care system using a systems-based approach.⁵ Curriculum was defined as a formal supervised program for changing knowledge, skills, or behaviors.¹¹ Clinician was defined as any practitioner who provides direct medical care to patients, excluding dentists and clinical laboratory workers. Studies were included if they described both a curriculum that teaches QI theory to clinicians and a comparative evaluation (ie, pre/post evaluations, time series, or randomized or nonrandomized controlled evaluation). Studies were excluded if they did not (1) occur in North America, Western Europe, Australia, New Zealand, or Japan; (2) teach QI theory; (3) describe a curricular intervention; (4) target clinicians; (5) include original data; (6) have a full article available for review; or (7) publish the results of a comparative evaluation. Studies also were excluded if they were published before 1980 or were not written in English.

These criteria were chosen because industrial process improvement methods first became popular in the 1980s (no eligible studies were published before 1995) and because we were primarily interested in the effects of QI curricula in the US health care system and other similarly developed medical systems. We included curricula for practicing physicians as well as for medical trainees because QI concepts should be relevant to both groups.

Title and Abstract Review

Eligible studies were identified through 2 phases. In the first phase, 2 authors (D.M.W. and L.S.F. or R.T.B. and C.C.) independently reviewed the titles of citations to assess eligibility for review. Citations were returned for adjudication if disagreement occurred. If agreement could not be reached, the citation was retained for further review.

In the second phase, the abstracts of remaining citations were independently reviewed by 2 pairs of reviewers using forms for determining eligibility criteria. If reviewers disagreed, the study article was returned for adjudication until agreement was reached. If eligibility could not be determined because an abstract had insufficient information, the full article was retrieved. Reviewers were not masked to author or journal name because previous work has shown that such masking is unlikely to make a significant difference in the results of data abstraction.¹²

Study Review

All remaining studies were retrieved for data abstraction. Two reviewer pairs (R.T.B. and C.C. or D.M.W. and L.S.F.) used a standardized form to confirm eligibility of the article. Reviewers resolved disagreements by consensus. Eligible studies were abstracted using a serial review process. Three reviewers divided the primary reviews and one reviewer checked all initial abstractions (R.T.B., L.S.F., C.C., and D.M.W.). Extracted information assessed the curricular context, characteristics of targeted learners, educational strategies, evaluation methods, and results. Quality improvement curricular content was based on the Institute for Healthcare Improvement (IHI) domains of knowledge for improving health care.¹³ The 8 IHI domains are a well-recognized framework for classifying QI knowledge.¹⁰ They include customer knowledge; health care as a system; variation and measurement; leading, following, and making change; collaboration; developing new, locally useful knowledge; social context and accountability; and professional knowledge. Educational content that overlapped with more than 1 domain was classified into all relevant domains. Studies also were classified according to whether they involved other types of noneducational QI interventions such as audit, feedback, or patient registries by adapting the Cochrane Effective Practice and Organization of Care taxonomy.¹⁴

We hypothesized that an association exists between using sound adult learning principles and curriculum effectiveness. To evaluate the use of adult learning principles, a list of fundamental principles was developed based on a review of 5 major educational theories: andragogy, social cognitive theory, self-directed learning, constructivism, and reflective practice.¹⁵ Using this list, curricula were assessed for whether they fully or partially exhibited 9 characteristics: (1) enabling learners to be active participants, (2) providing content relating to learners' current experiences, (3) assessing learners' needs and tailoring teaching to their past experiences, (4) allowing learners to identify and pursue their own learning goals, (5) allowing learners to practice their learning, (6) supporting learners during self-directed learning, (7) providing feedback to learners, (8) facilitating learner self-reflection, and (9) role-modeling behaviors. The reviewers relied on the authors' descriptions to determine if a principle was used. For example, a curriculum was classified as fully exhibiting facilitation of reflective learning if the article specifically described using learner self-reflection, or partially exhibiting that principle if it described features likely to encourage self-reflection, such as asking learners to discuss lessons learned.

The reliability of all categorical questions was assessed for the first 27 articles abstracted. Overall agreement was 91.5% and interrater reliability was good ( = 0.77) for data abstraction. Interrater reliability for assessing adult learning principles was moderate ( = 0.51).

Data Synthesis and Statistical Analysis

Due to the heterogeneity of results, outcomes were classified as beneficial effects, no effects, or detrimental effects based on whether differences from either baseline (in uncontrolled studies) or controls were statistically significant at the P value level of .05 or less. If the authors did not calculate statistical significance, an effect was counted as beneficial or detrimental if the absolute change or difference in an outcome measure was greater than 10%. We used 2-tailed ² statistics to assess differences in proportions. All statistical analyses were performed using Intercooled Stata version 9.2 (StataCorp, College Station, Texas).

RESULTS

Jump to Section  • Top  • Introduction  • Methods  • Results  • Comment  • Author information  • References

Characteristics of Eligible Studies

Of 16 278 citations identified, 39 studies met eligibility criteria (Figure). Four study articles described evaluations of 2 separate curricula.^16-19 Table 1 summarizes the characteristics of the studies. Eighty percent of studies were published in the last 5 years. All targeted physicians, medical trainees, nurses, or nursing students. Many curricula also taught other clinicians. Ten studies targeted trainees^20-29 and 29 targeted nontrainees.^16-19,30-54 Two of the IHI knowledge domains were covered by all of the curricula: leading, following, and making change; and developing new, locally useful knowledge. Nearly all of the studies described the use of didactic instruction in combination with experiential learning.^{16-29,31-33,35-54} Only 8 of the studies used a randomized controlled group design.^{16-17,31-32,35-36,40-41} Table 2 and Table 3 describe the characteristics of individual curricula, which were grouped into 3 categories to facilitate analysis: curricula for trainees, curricula for nontrainees that have an educational focus, and curricula for nontrainees that are included in a multifaceted QI intervention.

View larger version (52K): [in this window] [in a new window] [as a PowerPoint slide]   Figure. Literature Search and Study Selection Process

View this table: [in this window] [in a new window] [as a PowerPoint slide]   Table 1. Characteristics of Quality Improvement (QI) Curriculum Studies (N = 39)a

View this table: [in this window] [in a new window] [as a PowerPoint slide]   Table 2. Characteristics of Quality Improvement (QI) Curricula With Controlled Evaluations

View this table: [in this window] [in a new window] [as a PowerPoint slide]   Table 3. Characteristics of Quality Improvement (QI) Curricula With Uncontrolled Evaluations

Features of Curricula for Trainees

Of the 10 curricula targeting trainees, 3 involved either medical or nursing students.^26-27,29 Program details varied greatly: one involved preclinical nursing students observing patients' perspectives of hospital wards for 10 weeks before learning and discussing how to apply QI methods,²⁷ while another was a 4-year longitudinal program that integrated didactic instruction and QI activities into an existing medical school curriculum.²⁶

Seven studies involving residents had more similarities.^20-25,28 All took place during ambulatory care assignments or electives, all combined didactic instruction with participation in QI activities, and none included noneducational QI interventions. Five curricula were integrated into a 4-week rotation,^20-22,24-26 while 2 held weekly or biweekly meetings for a year.^{23, 28}

Features of Curricula for Nontrainees That Have an Educational Focus

Five of the 29 studies (17%) involving nontrainees described their primary focus as conveying QI concepts to learners.^30-34 Peters et al³³ described a 4-month medical school faculty development program teaching QI concepts. Irvine Doran et al³¹ taught QI concepts to hospital-based clinicians and encouraged application of these skills in their hospitals. The remaining 3 studies described brief (1- to 7-day) continuing education–style seminars.^{30, 32, 34}

Features of Curricula for Nontrainees That Are Included in a Multifaceted QI Intervention

Twenty-four of the 29 studies (83%) involving nontrainees were QI interventions with educational components.^16-19,35-54 All obtained administrative support for the intervention and provided faculty with QI expertise. We have grouped these studies into 4 subcategories according to characteristics of the overall QI intervention.

QI Training Session Supplementing Noneducational QI Interventions. Five studies described QI curricula integrated into larger QI interventions primarily managed by project organizers.^{17, 35-38} All taught QI concepts and methods during an initial training session associated with audit and feedback. Learners then implemented QI interventions with personalized faculty support. Most of these interventions also provided learners with a package of predeveloped QI instruments, such as patient educational materials or customized flow sheets.^{17, 35-37}

QI Curricula in IMPROVE and IDEAL Collaboratives. Nineteen of the eligible studies (49%) described teaching learners through QI collaboratives, whereby learners attended facilitated workshops in which they shared ideas and taught each other QI concepts related to a clinical process or disease focus. The Improving Prevention Through Organization, Vision and Empowerment (IMPROVE) and Improving Diabetes Care Through Empowerment, Active Collaboration and Leadership (IDEAL) collaboratives taught QI concepts to primary care teams focused on preventive care and diabetes management.^40-41 These curricula offered didactic instruction and interactive discussions during 7 half-day workshops over an 18- to 22-month period, interspersed with QI activities that were supported by faculty telephone calls and site visits. Instead of providing learners with a preexisting package of QI interventions, they focused on facilitating networking and sharing of ideas by learners, who implemented the ideas they believed to be most appropriate for their sites.

QI Curricula in Breakthrough Series Collaboratives. Twelve studies described curricula based on the IHI-sponsored Breakthrough Series.^{16, 18-19,42-49,54} Breakthrough Series collaboratives teach a QI approach using the chronic care model (a multidisciplinary approach to providing decision support and supporting patient self-management) and the model for improvement (a conceptual model for implementing and testing process change).⁵⁵ They require use of an electronic patient registry and have been used with more than 20 different conditions throughout the world. Similar to the IMPROVE and IDEAL programs, learners in Breakthrough Series curricula also attend multiple QI workshops over a 6- to 18-month period, interspersed with QI activities. However, Breakthrough Series collaboratives emphasize incremental small tests of change. For example, learners in one Breakthrough Series program implemented and tested an average of 42 small process changes over 12 months,⁴² in contrast to learners in an IDEAL collaborative who implemented and tested 1 comprehensive process change in 18 months.⁴⁰

QI Curricula in Interventions Using Other Collaborative Models. Five studies described QI curricula in the context of other collaboratives. Three were described as modifications of the Breakthrough Series model.^50-52 One used a unique conceptual model for teaching QI in the neonatal intensive care unit.⁵³ One supplemented QI training for coronary bypass graft surgery teams with observational site visits among learners' hospitals.³⁹ All 5 studies presented learners with audit and feedback at an initial learning session followed by teleconferences or site visits to facilitate collaboration during QI projects.

Use of Adult Learning Principles

The extent to which QI curricula used adult learning principles is presented in eTable 1. Most studies described more than 6 of the 9 adult learning principles. The studies involving trainees often provided richer descriptions of the educational intervention, and as a group they described the presence of more adult learning principles than those involving nontrainees (median number of principles 8 vs 6). Among the 10 studies involving trainees,^20-29 all curricula allowed learners to actively contribute to their education, engaged learners with topics that were relevant to their current experiences, helped learners direct their own learning, facilitated practice through QI activities, and described feedback strategies. Only 3 studies (30%) described role modeling as a teaching method.^{23, 26, 29}

The 29 studies of nontrainees described using adult learning principles less often than studies of trainees but more often than has been described for traditional continuing education.⁵⁶ All of these studies facilitated learners' active contributions and addressed topics relevant to their current experiences.^16-19,30-54 However, only 1 of these studies (3%) used role modeling,³³ only 6 studies (21%) described a needs assessment,^{17-18,33, 35, 42, 46} and only 18 studies (62%) reported giving feedback to learners.^{16-18,31, 35-39,43, 46-50,52-54}

Results of Evaluations

Because standard measures for assessing QI curricular effectiveness do not exist,⁵⁷ the studies in this review present heterogeneous outcomes (details are provided in eTable 2). Table 4 and Table 5 summarize the evaluation methods and direction of outcomes, organized by type of learner, type of intervention, and study design. Only 3 of the studies (8%) described both clinical and educational outcomes.^{22, 29, 48} Additionally, one study¹⁹ described behavioral changes related to clinical outcomes reported in another study.¹⁸

View this table: [in this window] [in a new window] [as a PowerPoint slide]   Table 4. Evaluation Methods and Direction of Educational Outcomes

View this table: [in this window] [in a new window] [as a PowerPoint slide]   Table 5. Evaluation Methods and Direction of Clinical Outcomes

With the exception of a single detrimental effect in learner attitudes,²⁹ all outcomes had either positive or null effects. Twelve studies (31%) reported only beneficial effects,^{17, 22-23,25, 27-28,30, 33-34,39, 43, 46} 24 studies (62%) reported mixed effects (positive, null, and detrimental),^{19-21,24, 26, 29, 31-32,35-38,41-42,44-45,47-54} and 3 studies found only null effects^{16, 18, 40} (Table 6). Studies most frequently reported improved knowledge and rarely found benefits in patient outcomes. Fewer controlled studies reported only positive outcomes compared with those using uncontrolled designs (4 of 22 controlled studies vs 8 of 17 uncontrolled studies; P = .05).

View this table: [in this window] [in a new window] [as a PowerPoint slide]   Table 6. Summary of Evaluation Results of Quality Improvement Curricula

Educational Outcomes

Of 14 studies describing educational outcomes,^{19, 21-24,26-27,29-34,48} 4 described curricula for residents^21-24; 3 for medical or nursing students^26-27,29; 1 for medical school faculty³³; and 6 for nontrainee clinicians.^{19, 30-32,34, 48} The direction of outcomes did not appear to differ by the training level of the learners.

Of the 10 studies (71%) that reported the effect of teaching QI on knowledge,^{21-22,24, 29-34,48} all 5 involving trainees reported improvement in either perceived knowledge^{24, 27} or scores on a QI knowledge test.^{21-22,24, 29} Four of 5 studies involving nontrainees found improvement in either perceived^{30, 33} or tested^{32, 34} knowledge. One study of hospital-based clinicians found no change in scores on a QI knowledge examination.³¹ Only 2 studies used the same knowledge test (Quality Improvement Knowledge Assessment Test). ^21-22

Five of 6 studies describing attitudinal changes involved trainees.^{21, 23-24,26, 29} One found residents' confidence in their ability to provide good diabetes care improved²³; another reported no change in residents' interest in QI.²⁴ One curriculum reported improved medical students' attitudes toward managed care,²⁶ while 2 studies found mixed attitudes toward QI interventions.^{21, 29} In these studies, learners reported improved confidence in their own or in their organization's ability to engage in QI projects but no change was seen in their confidence to sustain systemic change. Only 1 study described attitudinal changes in nontrainees. That study found that learners' confidence in their organization's system for diabetes management improved but their confidence in their organization's ability to support staff or community linkages did not change.⁴⁸

Six curricula evaluated changes in learners' skills or behaviors.^{19, 24, 30-33} The only program involving trainees found that residents' perceived ability to implement QI projects improved.²⁴ Of the 5 studies involving nontrainees, one curriculum for hospital-based teams reported improvement in functional group interactions using a validated scale,³¹ another curriculum for medical school faculty described improvement in their perceived ability to teach QI,³³ and 3 studies described improvement in self-reported ability to implement QI projects and greater involvement in QI activities.^{24, 30, 32} A final study found no changes in organizational behaviors for 4 of 6 chronic care model domains.¹⁹

Clinical Outcomes

The 5 curricula involving trainees that evaluated clinical outcomes generally reported positive results.^{20, 22, 25, 28-29} Most used uncontrolled evaluation methods with small sample sizes. Varkey et al²² reported 18% to 37% improvement in documentation processes before and after the QI interventions but did not assess statistical significance. Coleman et al²⁵ reported 24% to 34% improvement in documentation processes (P = .001) and 26% improvement in diabetic microalbuminuria screening (P = .02). Mohr et al²⁸ found significantly increased on-time chart availability (40%; P < .01), chart completeness (61%; P < .01), and immunization rates of infants (26%; P = .04). In a nonrandomized controlled study, Holmboe et al²⁰ found that diabetic patients of residents in the intervention group had 12% more annual monofilament examinations (P = .02), 10% more baseline electrocardiograms (P = .01), and greater decreases in levels of hemoglobin A_1c (1.1%; P = .001) and low-density lipoprotein cholesterol (20 mg/dL [0.52 mmol/L]; P = .01) than those followed up by control residents. No improvement, however, was seen in obtaining screening laboratory test results, providing immunizations, or patients' blood pressure.²⁰ Gould et al²⁹ described the only trainee-centered curriculum that integrated noneducational QI interventions, including chart audit tools, and patient education materials. The documentation of diabetic eye and foot examinations and patients' hemoglobin A_1c levels improved significantly, although rates of documenting levels of hemoglobin A_1c (which was high at baseline) did not change.²⁹

All curricula that involved nontrainees and reported clinical outcomes were associated with other noneducational QI interventions. They reported a diverse set of outcomes, individually reporting a mean of 9.7 (range, 1-25) different clinical outcomes. The results of these studies were mixed and are detailed in eTable 2. The key outcomes of controlled studies are described herein.

Most studies that supplemented traditional audit and feedback with individual coaching in QI methods reported positive process outcomes and highlighted the importance of in-person faculty coaching.^{17, 35-38} One curriculum—described in 2 separate studies^{17, 35}—randomized pediatric practices to receive audit and feedback with either cursory instruction in process improvement methods or extensive coaching in QI methods. Margolis et al¹⁷ found higher adjusted odds of providing 4 preventive services in interventional practices than in control practices (odds ratio, 4.6; 95% confidence interval [CI], 1.6-13.2). Rosenthal et al³⁵ reported that rates of all anticipatory counseling measures increased 18.7% to 22.6% compared with controls (P = .002 to P = .001) but found no differences in parents' health knowledge or behaviors. Similar benefits were seen in 2 uncontrolled nursing home studies that used similar teaching methods.^37-38 In contrast, McClellan et al³⁶ found that adding didactic QI training and optional telephone-based faculty support to traditional audit and feedback was associated with improvement in only 4 of 10 measures of hemodialysis quality.

Evaluations of the IMPROVE and IDEAL collaborative models reported mostly null findings.^40-41 Although Solberg et al⁴¹ found increased rates of pneumococcal immunizations and offers for cholesterol screening compared with controls, the authors found no significant differences in 10 other preventive care measures. Similarly, O’Connor et al⁴⁰ reported no significant improvement compared with controls in 12 diabetes process measures.

By comparison, Breakthrough Series collaboratives^{16, 18, 42-49,54} reported more positive outcomes. Among the controlled studies, 1 reported only beneficial results for 5 measures of adherence to cardiac care guidelines,⁴³ and 3 found significant improvements in 26 of 51 process outcomes and 4 of 12 patient outcomes related to asthma, cardiovascular care, and diabetes care.^{42, 44-45} However, not all Breakthrough Series curricula reported beneficial effects. Landon et al¹⁸ found no improvement in 6 process measures related to the human immunodeficiency virus (HIV) or in the proportion of controlled viral loads. Homer et al¹⁶ reported no change in use of asthma action plans or in the patients' symptoms and suggested that incomplete learner attendance may have influenced their findings. McInnes et al¹⁹ surveyed clinicians and administrators from the HIV collaborative described by Landon et al¹⁸ and theorized that the absence of organizational changes in 4 of 6 domains of the chronic care model may explain their null findings.

In contrast to QI collaboratives held in outpatient settings, those in highly controlled inpatient settings reported beneficial effects for a focused set of outcomes.^{39, 50-53} One curriculum involving neonatal intensive care units reported significantly higher adjusted odds of on-time surfactant delivery in the intervention group compared with controls who received only audit and feedback (odds ratio, 5.38; 95% CI, 2.84-10.20).⁵³ Similarly, O’Connor et al³⁹ reported a significantly lower standardized mortality ratio for cardiothoracic surgery at interventional hospitals compared with historical controls (standardized mortality ratio, 0.76; 95% CI, 0.67-0.90).

Association Between Use of Adult Learning Principles and Curriculum Outcomes

An exploratory analysis was performed to assess the relationship between use of adult learning principles and curriculum effectiveness by comparing the proportion of curricula reporting only beneficial effects with the number of adult learning principles described. Curricula with only beneficial effects, mixed effects, and only null effects all described a median of 7 principles (range 2-8). No relationships with this measure were seen using only controlled studies or those with only educational outcomes.

Analyses also were performed to assess if relationships exist between clinical outcomes and learner attendance, as well as between clinical and educational outcomes. Eight studies described incomplete attendance.^{16, 18, 25, 37, 41, 47-48,51} Among these, all 4 controlled evaluations found either no benefit or benefit in less than 25% of described outcomes.^{16, 18, 41, 51} Results for both educational and clinical evaluations were only available for 4 curricula. One reported improvements in both residents' QI knowledge and documentation processes,²² and another found improvements in practitioners' attitudes and in their patients' hemoglobin A_1c levels.⁴⁸ Neither study evaluated the statistical significance of their clinical outcomes. Gould et al²⁹ reported increased knowledge and confidence in QI methods for 26 of 40 survey items and statistically significant improvement in diabetes-related processes and outcomes at medical students' preceptor sites. In contrast, the null clinical findings of an HIV collaborative were linked to an absence of change in organizational behaviors.^18-19

Resources Used by Curricula

Only 6 studies described QI curricular costs.^{20-21,41, 45-46,48} In one program, residents spent an average of 119 hours in 4 weeks on their projects, faculty spent 6 hours per month teaching the curriculum, and project sponsors devoted 1 to 2 hours per week supporting residents' activities.²¹ These findings were comparable with other studies that reported time costs. Holmboe et al²⁰ found that residents committed 4 half days and faculty committed 3 hours over 4 weeks, while Solberg et al⁴¹ reported that practicing clinicians spent 5 to 11.7 hours per month on their projects.

Monetary costs varied greatly and likely reflected resources needed to integrate QI curricula with other noneducational QI interventions. Ogrinc et al²¹ described the direct costs of their residency-based curriculum as negligible, whereas health care organizations each paid $12 500 plus travel expenses to enroll a 3-person team in a Breakthrough Series curriculum.^45-46,48

COMMENT

Jump to Section  • Top  • Introduction  • Methods  • Results  • Comment  • Author information  • References

To our knowledge, this is the first systematic review to evaluate the effect of teaching QI to clinicians. The 39 studies in our sample varied in their coverage of topics within the IHI domains of QI knowledge. All described teaching content required for implementing and evaluating local systems changes but few taught learners about the larger health care system or how to view the system from a patient's perspective.

Fourteen studies evaluating educational outcomes collectively suggest that using a variety of QI teaching strategies improves QI-related knowledge, as well as attitudes toward health systems and participation in QI activities for learners of many training levels and professions. However, relatively weak study designs and evaluation instruments hamper these assessments. Most educational evaluations use uncontrolled designs, which are vulnerable to secular trends and other biases. Furthermore, those that report changes in knowledge often rely on learners' self-report of knowledge, and the validity of this assessment method has been questioned.⁵⁸ Of those using knowledge tests, only 2 studies use a validated test.^21-22 The validity of learner attitudinal and behavioral assessments is similarly uncertain.

Curricula with clinical outcomes provide mixed evidence for improvement in processes and little evidence for beneficial patient outcomes. Those involving trainees report mostly beneficial outcomes in documentation and disease management processes but use relatively weak study designs.

All nontrainee curricula with clinical outcomes were components of multifaceted QI interventions that provided expert faculty and received administrative support. Of these, the IMPROVE and IDEAL programs placed greatest emphasis on theoretical QI content and included the fewest external interventions. Although their teaching strategies applied many adult learning principles, they nevertheless found little evidence for clinical benefit compared with nonparticipant controls. Their null findings stand in contrast to the more positive findings of programs that combined QI training with noncollaborative QI interventions. From these studies, it appears that providing learners with a package of QI tools and individualized coaching in QI methods can improve clinical outcomes compared with audit and feedback and limited QI training.

The positive outcomes associated with these programs and with the Breakthrough Series curricula also suggest that providing learners with access to preexisting performance data or process improvement tools can improve outcomes. For example, the IMPROVE and IDEAL curricula encouraged learners to collect their own performance data, whereas curricula in the Breakthrough Series collaboratives and those supplementing noncollaborative QI interventions provided learners with data from an electronic registry or external audits. The positive outcomes seen in curricula set in data-rich inpatient and intensive care environments also support the importance of access to performance data. Similarly, many curricula with positive outcomes also provided learners with preexisting decision support or patient education tools, which may have allowed them to focus on implementing interventions instead of developing their own tools. The relatively greater benefits seen in Breakthrough Series curricula also suggest that encouraging learners to implement and test interventions through several small cycles of trial and error may result in better outcomes than undertaking a single comprehensive intervention.

Nevertheless, the null findings in studies of HIV and pediatric Breakthrough Series collaboratives indicate that performing incremental tests of change and having ready access to performance data are insufficient to ensure beneficial clinical outcomes. The attendance issues discussed by Homer et al¹⁶ and the absence of organizational changes following the HIV collaborative^18-19 may indicate that how well clinicians learned QI concepts influences clinical outcomes. The findings of 4 curricula for which educational and clinical outcomes are available^{18-19,22, 29, 48} suggest that clinical benefits occur when educational outcomes improve and are absent when learners’ behaviors do not change. Although our analysis did not find convincing evidence for an association between the use of adult learning principles and curriculum effectiveness, we were limited to the details that the authors published and we could not reliably assess important details such as attendance, learners' prior experiences, faculty QI expertise, or faculty teaching skill. Furthermore, counting curricula with only beneficial, mixed, or only null effects provides a crude measure of curriculum effectiveness.

This review has several limitations. First, it is subject to publication bias. Although we could not formally test for this, we did examine the sample size distribution for studies that had positive, mixed, and null effects, and found a relatively similar sample size distribution for each group, arguing against the presence of major publication bias. Second, our findings may not apply to medical education in all countries because we limited our review to studies that were published in English and occurred in countries with health care systems similar to the United States. In doing so, we inadvertently excluded countries such as Korea, Singapore, and Taiwan, which also have highly developed health care systems. However, in a retrospective review of the citations we excluded, we found no studies from those countries that would have met our eligibility criteria. Third, we focused on curricula that described results with comparisons to either baseline results or with controls, and thus excluded several descriptions of innovative QI curricula.^59-63 We did so because we wished to examine the effects of teaching QI concepts instead of trying to describe the variety of ways in which they have been taught. Similarly, we excluded studies about QI interventions that did not provide at least an adequate description of the curriculum because we hoped to determine the effect of educational methods on curriculum effectiveness. Also, our analysis of the relationship between use of adult learning principles and curriculum effectiveness should be considered exploratory because no validated instrument exists for assessing the use of adult learning principles. Finally, our ability to combine outcomes was limited by their heterogeneity, which is an ongoing issue in systematic reviews of educational and organizational interventions in health care.^64-65

Despite these limitations, we offer some recommendations for developing and evaluating future QI curricula. Published QI curricula teach methods for implementing systems change but infrequently address the IHI knowledge domains of health care as a system and customer knowledge. Because there was little evidence for improvement in patient knowledge or behaviors despite improvements in several measures of counseling processes, we recommend that future QI curricula also cover these IHI-defined domains. To advance the understanding of how teaching QI influences educational and clinical outcomes, educators should describe key features of their curricula, including learner and faculty characteristics, attendance, and learners' interventions. They also should compare learners with controls and present both educational and clinical outcomes. Part of the attractiveness of teaching QI to clinicians is that learners will theoretically expand process improvement efforts independently.^{3, 66} Thus, future curricula may be evaluated based on whether learners' attitudes, knowledge, and skills improve, particularly if improvements are associated with intermediate clinical gains. Although past efforts to assess educational outcomes in QI have been hampered by the lack of assessment tools, we did find examples of potentially useful instruments for assessing QI-related knowledge and skills.^{21, 67-68}

We also recommend that future curricula teach collaborative skills, facilitate experiential learning with incremental change from trial and error, and provide learners with opportunities to work closely with colleagues from other disciplines. These are accepted principles of adult learning, and collaborative small tests of change appear to be important for achieving clinical benefit. Although evidence suggests that intensive coaching from expert faculty improves clinical outcomes, few educators currently have QI expertise. Furthermore, it is important to balance the desire for improved processes with the need for clinicians to learn through trial and error. Quality improvement curricula should provide learners with access to resources that facilitate their QI activities, such as performance data and predeveloped process improvement tools, because evidence suggests that these resources may enhance the clinical effectiveness of teaching QI. Although such curricula can be complex, time-consuming, and expensive, we found several examples of curricula that exhibited these features and were successfully incorporated into busy training programs.^{20-21,24, 29}

AUTHOR INFORMATION

Jump to Section  • Top  • Introduction  • Methods  • Results  • Comment  • Author information  • References

Corresponding Author: Romsai T. Boonyasai, MD, MPH, Collaborative Internal Medicine Service, Johns Hopkins Bayview Medical Center, 5200 Eastern Ave, West Tower, Sixth Floor, Baltimore, MD 21224 (romsai{at}jhmi.edu).

Author Contributions: Dr Boonyasai had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Boonyasai, Windish, Chakraborti, Rubin, Bass.

Acquisition of data: Boonyasai, Windish, Chakraborti, Feldman.

Analysis and interpretation of data: Boonyasai, Feldman, Rubin, Bass.

Drafting of the manuscript: Boonyasai.

Critical revision of the manuscript for important intellectual content: Boonyasai, Windish, Chakraborti, Feldman, Rubin, Bass.

Statistical analysis: Boonyasai, Chakraborti.

Obtained funding: Rubin.

Study supervision: Rubin, Bass.

Financial Disclosures: None reported.

Funding/Support: The authors are members of the Quality Improvement Curriculum Committee appointed by the Society of General Internal Medicine, which provided small honoraria to Drs Boonyasai, Windish, Chakraborti, and Feldman, as well as support for a portion of the time spent by research staff. Dr Boonyasai was supported by a National Heart, Lung, and Blood Institute T32 training grant (HL07180). Dr Chakraborti was supported by National Research Service Award-Health Resources and Services Administration fellowship training grant (5-T-32-HP10025). Drs Bass and Feldman receive support from the Osler Center for Clinical Excellence at Johns Hopkins University.

Role of the Sponsor: The funding organizations and professional societies had no input in the design and conduct of this study; in the collection, analysis, and interpretation of the data; or in the preparation, review or approval of the manuscript.

Additional Contributions: Renee Wilson, MS (Johns Hopkins University Evidence-Based Practice Center, Baltimore, Maryland), and Cindy Sheffield, MBA, MLS (Welch Medical Library, and Johns Hopkins University, Baltimore), provided assistance in performing the literature search and using the data management software. Haeseong Park, MD, MPH (Johns Hopkins University Evidence-Based Practice Center and Bloomberg School of Public Health), provided assistance with hand searching. Ira Wilson, MD, MSc (Tufts-New England Medical Center, Boston, Massachusetts), provided intellectual support for this review. None of the persons listed received compensation for their assistance with this project.

Author Affiliations: Department of Internal Medicine, Johns Hopkins School of Medicine, Baltimore, Maryland (Drs Boonyasai, Feldman, and Bass); Department of Internal Medicine, Yale University School of Medicine, New Haven, Connecticut (Dr Windish); Department of Internal Medicine, George Washington University School of Medicine, Washington, DC (Dr Chakraborti); and Palo Alto Medical Foundation Research Institute, Palo Alto, California (Dr Rubin).

REFERENCES

Jump to Section  • Top  • Introduction  • Methods  • Results  • Comment  • Author information  • References

1. American Association of Colleges of Nursing. The Essentials of Master's Education for Advanced Practice Nursing. Washington, DC: American Association of Colleges of Nursing; 1996:6-9. 2. Institute of Medicine. Crossing the Quality Chasm. Washington, DC: National Academy Press; 2001:207-214.3. Berwick DM. Continuous improvement as an ideal in health care. N Engl J Med. 1989;320(1):53-56. WEB OF SCIENCE | PUBMED 4. Headrick LA, Neuhauser D, Schwab P, Stevens DP. Continuous quality improvement and the education of the generalist physician. Acad Med. 1995;70(1)(suppl):S104-S109. WEB OF SCIENCE | PUBMED 5. Institute of Medicine; Committee on Health Professions Education Summit. Health Professions Education: A Bridge to Quality. Washington, DC: National Academy Press; 2003:45-96.6. Accreditation Council on Graduate Medical Education Outcome Project Web site. http://www.acgme.org/outcome/comp/compFull.asp. Accessed May 9, 2006.7. Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ. 1995;153(10):1423-1431. ABSTRACT 8. Gilbody S, Whitty P, Grimshaw J, Thomas R. Educational and organizational interventions to improve the management of depression in primary care: a systematic review. JAMA. 2003;289(23):3145-3151. FREE FULL TEXT 9. Evans D, Sheares BJ, Vazquez TL. Educating health professionals to improve quality of care for asthma. Paediatr Respir Rev. 2004;5(4):304-310. FULL TEXT | PUBMED 10. Ogrinc G, Headrick LA, Mutha S, Coleman MT, O'Donnell J, Miles PV. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review. Acad Med. 2003;78(7):748-756. WEB OF SCIENCE | PUBMED 11. Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development for Medical Education: A Six-Step Approach. Baltimore, MD: Johns Hopkins University Press; 1998:1.12. Berlin JA, University of Pennsylvania Meta-analysis Blinding Study Group. Does blinding of readers affect the results of meta-analyses? Lancet. 1997;350(9072):185-186. WEB OF SCIENCE | PUBMED 13. Batalden P, Berwick DM, Bisognano M, Splaine M, Baker G, Headrick L. Knowledge Domains for Health Professional Students Seeking Competency in the Continual Improvement and Innovation of Health Care. Boston, MA: Institute for Healthcare Improvement; 1998:1-2.14. Shojania KG, McDonald KM, Wachter RM, Owens DK. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Vol 1. Rockville, MD: Agency for Healthcare Research and Quality; 2006:14-18.15. Kaufman DM. Applying educational theory in practice. BMJ. 2003;326(7382):213-216. FREE FULL TEXT 16. Homer CJ, Forbes P, Horvitz L, Peterson LE, Wypij D, Heinrich P. Impact of a quality improvement program on care and outcomes for children with asthma. Arch Pediatr Adolesc Med. 2005;159(5):464-469. FREE FULL TEXT 17. Margolis PA, Lannon CM, Stuart JM, Fried BJ, Keyes-Elstein L, Moore DEJ. Practice based education to improve delivery systems for prevention in primary care: randomised trial. BMJ. 2004;328(7436):388. FREE FULL TEXT 18. Landon BE, Wilson IB, McInnes K, et al. Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study. Ann Intern Med. 2004;140(11):887-896. FREE FULL TEXT 19. McInnes DK, Landon BE, Wilson IB, et al. The impact of a quality improvement program on systems, processes, and structures in medical clinics. Med Care. 2007;45(5):463-471. FULL TEXT | WEB OF SCIENCE | PUBMED 20. Holmboe ES, Prince L, Green M. Teaching and improving quality of care in a primary care internal medicine residency clinic. Acad Med. 2005;80(6):571-577. FULL TEXT | WEB OF SCIENCE | PUBMED 21. Ogrinc G, Headrick LA, Morrison LJ, Foster T. Teaching and assessing resident competence in practice-based learning and improvement. J Gen Intern Med. 2004;19(5 pt 2):496-500. FULL TEXT | WEB OF SCIENCE | PUBMED 22. Varkey P, Reller MK, Smith A, Ponto J, Osborn M. An experiential interdisciplinary quality improvement education initiative. Am J Med Qual. 2006;21(5):317-322. FREE FULL TEXT 23. Nuovo J, Balsbaugh T, Barton S, et al. Development of a diabetes care management curriculum in a family practice residency program. Dis Manage. 2004;7(4):314-324. FULL TEXT 24. Djuricich AM, Ciccarelli M, Swigonski NL. A continuous quality improvement curriculum for residents. Acad Med. 2004;79(10)(suppl):S65-S67. FULL TEXT | WEB OF SCIENCE | PUBMED 25. Coleman MT, Nasraty S, Ostapchuk M, Wheeler S, Looney S, Rhodes S. Introducing practice-based learning and improvement ACGME core competencies into a family medicine residency curriculum. Jt Comm J Qual Saf. 2003;29(5):238-247. PUBMED 26. O'Connell MT, Rivo ML, Mechaber AJ, Weiss BA. A curriculum in systems-based care. Fam Med. 2004;36(suppl):S99-S104. PUBMED 27. Kyrkjebø JM, Hanssen TA, Haugland BO. Introducing quality improvement to pre-qualification nursing students. Qual Health Care. 2001;10(4):204-210. FREE FULL TEXT 28. Mohr JJ, Randolph GD, Laughon MM, Schaff E. Integrating improvement competencies into residency education. Ambul Pediatr. 2003;3(3):131-136. FULL TEXT | WEB OF SCIENCE | PUBMED 29. Gould BE, Grey MR, Huntington CG, et al. Improving patient care outcomes by teaching quality improvement to medical students in community-based practices. Acad Med. 2002;77(10):1011-1018. WEB OF SCIENCE | PUBMED 30. François P, Vinck D, Labarere J, Reverdy T, Peyrin JC. Assessment of an intervention to train teaching hospital care providers in quality management. Qual Saf Health Care. 2005;14(4):234-239. FREE FULL TEXT 31. Irvine Doran DM, Baker GR, Murray M, et al. Achieving clinical improvement: an interdisciplinary intervention. Health Care Manage Rev. 2002;27(4):42-56. WEB OF SCIENCE | PUBMED 32. Saturno PJ. Training health professionals to implement quality improvement activities. Int J Qual Health Care. 1995;7(2):119-126. FREE FULL TEXT 33. Peters AS, Ladden MD, Kotch JB, Fletcher RH. Evaluation of a faculty development program in managing care. Acad Med. 2002;77(11):1121-1127. WEB OF SCIENCE | PUBMED 34. Palumbo G, Morosini P, Picardi A, Gigantesco A, Geddes J. Educating mental health professionals in clinical epidemiology and continuous quality improvement. Adm Policy Ment Health. 2004;31(5):421-424. FULL TEXT | WEB OF SCIENCE | PUBMED 35. Rosenthal MS, Lannon CM, Stuart JM, Brown L, Miller WC, Margolis PA. A randomized trial of practice-based education to improve delivery systems for anticipatory guidance. Arch Pediatr Adolesc Med. 2005;159(5):456-463. FREE FULL TEXT 36. McClellan WM, Hodgin E, Pastan S, McAdams L, Soucie M. A randomized evaluation of two health care quality improvement program (HCQIP) interventions to improve the adequacy of hemodialysis care of ESRD patients. J Am Soc Nephrol. 2004;15(3):754-760. FREE FULL TEXT 37. Horner JK, Hanson LC, Wood D, Silver AG, Reynolds KS. Using quality improvement to address pain management practices in nursing homes. J Pain Symptom Manage. 2005;30(3):271-277. FULL TEXT | WEB OF SCIENCE | PUBMED 38. Hanson LC, Reynolds K, Henderson M, Pickard C. A quality improvement intervention to increase palliative care in nursing homes. J Palliat Med. 2005;8(3):576-584. FULL TEXT | PUBMED 39. O'Connor GT, Plume SK, Olmstead EM, et al, Northern New England Cardiovascular Disease Study Group. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. JAMA. 1996;275(11):841-846. FREE FULL TEXT 40. O'Connor PJ, Desai J, Solberg LI, et al. Randomized trial of quality improvement intervention to improve diabetes care in primary care settings. Diabetes Care. 2005;28(8):1890-1897. FREE FULL TEXT 41. Solberg LI, Kottke TE, Brekke ML, et al. Failure of a continuous quality improvement intervention to increase the delivery of preventive services. Eff Clin Pract. 2000;3(3):105-115. PUBMED 42. Landon BE, Hicks LS, O'Malley AJ, et al. Improving the management of chronic disease at community health centers. N Engl J Med. 2007;356(9):921-934. FREE FULL TEXT 43. Peterson A, Carlhed R, Lindahl B, et al. Improving guideline adherence through intensive quality improvement and the use of a National Quality Register in Sweden for acute myocardial infarction. Qual Manag Health Care. 2007;16(1):25-37. PUBMED 44. Mangione-Smith R, Schonlau M, Chan KS, et al. Measuring the effectiveness of a collaborative for quality improvement in pediatric asthma care. Ambul Pediatr. 2005;5(2):75-82. FULL TEXT | WEB OF SCIENCE | PUBMED 45. Asch SM, Baker DW, Keesey JW, et al. Does the collaborative model improve care for chronic heart failure? Med Care. 2005;43(7):667-675. FULL TEXT | WEB OF SCIENCE | PUBMED 46. Boushon B, Provost L, Gagnon J, Carver P. Using a virtual breakthrough series collaborative to improve access in primary care. Jt Comm J Qual Patient Saf. 2006;32(10):573-584. PUBMED 47. Chin MH, Cook S, Drum ML, et al. Improving diabetes care in midwest community health centers with the health disparities collaborative. Diabetes Care. 2004;27(1):2-8. FREE FULL TEXT 48. Wagner EH, Glasgow RE, Davis C, et al. Quality improvement in chronic illness care. Jt Comm J Qual Improv. 2001;27(2):63-80. PUBMED 49. Weeks WB, Mills PD, Waldron J, Brown SH, Speroff T, Coulson LR. A model for improving the quality and timeliness of compensation and pension examinations in VA facilities. J Healthc Manag. 2003;48(4):252-262. 50. Young PC, Glade GB, Stoddard GJ, Norlin C. Evaluation of a learning collaborative to improve the delivery of preventive services by pediatric practices. Pediatrics. 2006;117(5):1469-1476. FREE FULL TEXT 51. Colón-Emeric C, Schenck A, Gorospe J, et al. Translating evidence-based falls prevention into clinical practice in nursing facilities. J Am Geriatr Soc. 2006;54(9):1414-1418. FULL TEXT | WEB OF SCIENCE | PUBMED 52. Baier RR, Gifford DR, Patry G, et al. Ameliorating pain in nursing homes. J Am Geriatr Soc. 2004;52(12):1988-1995. FULL TEXT | WEB OF SCIENCE | PUBMED 53. Horbar JD, Carpenter JH, Buzas J, et al. Collaborative quality improvement to promote evidence based surfactant for preterm infants. BMJ. 2004;329(7473):1004. FREE FULL TEXT 54. Stoeckle-Roberts S, Reeves MJ, Jacobs BS, et al. Closing gaps between evidence-based stroke care guidelines and practices with a collaborative quality improvement project. Jt Comm J Qual Patient Saf. 2006;32(9):517-527. PUBMED 55. Institute for Healthcare Improvement. The Breakthrough Series: IHI's Collaborative Model for Achieving Breakthrough Improvement. Cambridge, MA: Institute for Healthcare Improvement; 2003:1-20.56. Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of Continuing Medical Education. Rockville, MD: Agency for Healthcare Research and Quality; 2007:28.57. Mittman BS. Creating the evidence base for quality improvement collaboratives. Ann Intern Med. 2004;140(11):897-901. FREE FULL TEXT 58. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence. JAMA. 2006;296(9):1094-1102. FREE FULL TEXT 59. Moore SM, Alemi F, Headrick LA, et al. Using learning cycles to build an interdisciplinary curriculum in CI for health professions students in Cleveland. Jt Comm J Qual Improv. 1996;22(3):165-171. PUBMED 60. Headrick LA, Moore SM, Alemi F, et al. Using PDSA (Plan-Do-Study-Act) to establish academic-community partnerships. Qual Manag Health Care. 1998;6(2):12-20. PUBMED 61. Ellrodt AG. Introduction of total quality management (TQM) into an internal medicine residency. Acad Med. 1993;68(11):817-823. WEB OF SCIENCE | PUBMED 62. Ladden MD, Bednash G, Stevens DP, Moore GT. Educating interprofessional learners for quality, safety and systems improvement. J Interprof Care. 2006;20(5):497-505. FULL TEXT | PUBMED 63. Weingart SN, Tess A, Driver J, Aronson MD, Sands K. Creating a quality improvement elective for medical house officers. J Gen Intern Med. 2004;19(8):861-867. FULL TEXT | WEB OF SCIENCE | PUBMED 64. Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK. Challenges in systematic reviews. Ann Intern Med. 2005;142(12 pt 2):1056-1065. FREE FULL TEXT 65. Reed D, Price EG, Windish DM, et al. Challenges in systematic reviews of educational intervention studies. Ann Intern Med. 2005;142(12 pt 2):1080-1089. FREE FULL TEXT 66. Laffel G, Blumenthal D. The case for using industrial quality management science in health care organizations. JAMA. 1989;262(20):2869-2873. FREE FULL TEXT 67. Watson WE, Michaelsen LK, Sharp W. Member competence, group interaction, and group decision making: a longitudinal study. J Appl Psychol. 1991;76(6):803-809. FULL TEXT | WEB OF SCIENCE 68. Varkey P, Natt N. The Objective Structured Clinical Examination as an educational tool in patient safety. Jt Comm J Qual Patient Saf. 2007;33(1):48-53. PUBMED

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

RELATED ARTICLE

Teaching Quality Improvement: The Devil Is in the Details
Paul Batalden and Frank Davidoff
JAMA. 2007;298(9):1059-1061.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Quality: The Mayo Clinic Approach
Swensen et al.
American Journal of Medical Quality 2009;24:428-440.
ABSTRACT  

Improvement education by improving education: a model for integration of teaching, learning and research while practice is improved
Andersson-Gare
Qual Saf Health Care 2009;18:246-247.
FULL TEXT  

A hospital-randomized controlled trial of a formal quality improvement educational program in rural and small community Texas hospitals: one year results
Filardo et al.
Int J Qual Health Care 2009;21:225-232.
ABSTRACT | FULL TEXT  

Potential of Integrated Continuous Surveys and Quality Management to Support Monitoring, Evaluation, and the Scale-Up of Health Interventions in Developing Countries
Rowe
Am J Trop Med Hyg 2009;80:971-979.
ABSTRACT | FULL TEXT  

Doctors and quality improvement
Hockey and Marshall
JRSM 2009;102:173-176.
FULL TEXT  

Quality Improvement: Science and Action
Ting et al.
Circulation 2009;119:1962-1974.
FULL TEXT  

The Impact of a Preventive Cardiology Quality Improvement Intervention on Residents and Clinics: A Qualitative Exploration
Bernabeo et al.
American Journal of Medical Quality 2009;24:99-107.
ABSTRACT  

Continuing Medical Education Effect on Clinical Outcomes: Effectiveness of Continuing Medical Education: American College of Chest Physicians Evidence-Based Educational Guidelines
Mazmanian et al.
Chest 2009;135:49S-55S.
ABSTRACT | FULL TEXT  

Clinicians in Quality Improvement: A New Career Pathway in Academic Medicine
Shojania and Levinson
JAMA 2009;301:766-768.
FULL TEXT  

ACMQ Launches Its New Comprehensive Textbook on Medical Quality Management
Varkey and Diamond
American Journal of Medical Quality 2009;24:77-78.
 

Quality improvement in a publicly provided long-term care system: the case of Norway
Kjos et al.
Int J Qual Health Care 2008;20:433-438.
ABSTRACT | FULL TEXT  

Doctors, managers and the battle for quality
Marshall
JRSM 2008;101:330-331.
FULL TEXT  

Accelerating Best Care in Pennsylvania: Adapting a Large Academic System's Quality Improvement Process to Rural Community Hospitals
Haydar et al.
American Journal of Medical Quality 2008;23:252-258.
ABSTRACT  

Assessing Competencies of Knowledge and Process Improvement--Reply
Holmboe et al.
JAMA 2008;299:2276-2277.
FULL TEXT  

A proposal: publication guidelines for healthcare improvement education reports
Stevens and Andersson-Gare
Qual Saf Health Care 2007;16:402-402.
FULL TEXT  

Teaching Quality Improvement: The Devil Is in the Details
Batalden and Davidoff
JAMA 2007;298:1059-1061.
FULL TEXT