Effects of Computer-Based Clinical Decision Support Systems on Physician Performance and Patient Outcomes
A Systematic Review
- Dereck L. Hunt, MD;
- R. Brian Haynes, MD, PhD;
- Steven E. Hanna, MA, PhD;
- Kristina Smith
- From the Graduate Program in Health Research Methodology (Dr Hunt), the Health Information Research Unit, Department of Clinical Epidemiology and Biostatistics (Drs Hunt, Haynes, and Hanna and Ms Smith), and the Department of Medicine (Drs Hunt and Haynes), McMaster University Faculty of Health Sciences, Hamilton, Ontario.
Abstract
Context.— Many computer software developers and vendors claim that their systems can directly improve clinical decisions. As for other health care interventions, such claims should be based on careful trials that assess their effects on clinical performance and, preferably, patient outcomes.
Objective.— To systematically review controlled clinical trials assessing the effects of computer-based clinical decision support systems (CDSSs) on physician performance and patient outcomes.
Data Sources.— We updated earlier reviews covering 1974 to 1992 by searching the MEDLINE, EMBASE, INSPEC, SCISEARCH, and the Cochrane Library bibliographic databases from 1992 to March 1998. Reference lists and conference proceedings were reviewed and evaluators of CDSSs were contacted.
Study Selection.— Studies were included if they involved the use of a CDSS in a clinical setting by a health care practitioner and assessed the effects of the system prospectively with a concurrent control.
Data Extraction.— The validity of each relevant study (scored from 0-10) was evaluated in duplicate. Data on setting, subjects, computer systems, and outcomes were abstracted and a power analysis was done on studies with negative findings.
Data Synthesis.— A total of 68 controlled trials met our criteria, 40 of which were published since 1992. Quality scores ranged from 2 to10, with more recent trials rating higher (mean, 7.7) than earlier studies (mean, 6.4) (P<.001). Effects on physician performance were assessed in 65 studies and 43 found a benefit (66%). These included 9 of 15 studies on drug dosing systems, 1 of 5 studies on diagnostic aids, 14 of 19 preventive care systems, and 19 of 26 studies evaluating CDSSs for other medical care. Six of 14 studies assessing patient outcomes found a benefit. Of the remaining 8 studies, only 3 had a power of greater than 80% to detect a clinically important effect.
Conclusions.— Published studies of CDSSs are increasing rapidly, and their quality is improving. The CDSSs can enhance clinical performance for drug dosing, preventive care, and other aspects of medical care, but not convincingly for diagnosis. The effects of CDSSs on patient outcomes have been insufficiently studied.
- KEYWORDS:
- computer-assisted decision making
- decision support systems, clinical
- decision making, computer-assisted
- outcome assessment (health care)
COMPUTER SYSTEMS have long been promoted for their potential to improve the quality of health care, including their use to support clinical decisions.1 Access from clinical settings to bibliographic databases, such as MEDLINE and electronic versions of medical journals and textbooks, is becoming commonplace. Most of these resources, however, leave to the practitioner the task of matching individual patient characteristics to information. Computer-based decision support systems (CDSSs) provide additional assistance. They can synthesize and integrate patient-specific information, perform complex evaluations, and present the results to clinicians in a timely fashion.2 Examples include CDSSs designed to recommend appropriate drug doses, to provide immunization reminders, or to diagnose the cause of a patient's chest pain.3 As with any innovative health care intervention, however, these systems should be rigorously evaluated before widespread introduction into clinical practice. The various stages in this assessment process have been described by Wyatt and Spiegelhalter,4 including using controlled trials to test the effects on clinical behavior and patient outcomes.
Our group previously reviewed controlled trials of computer-aided quality assurance5 and CDSSs published up to 1992.3 This field is rapidly evolving because of technological advances and increasing access to computer systems in clinical practice. Thus, we updated the previous review to provide a cumulative picture of the state of documentation of the effects CDSSs have on clinician performance and patient outcomes based on the strongest trials in health care settings to date.
METHODS
Study Identification and Selection
A CDSS was defined as any software designed to directly aid in clinical decision making in which characteristics of individual patients are matched to a computerized knowledge base for the purpose of generating patient-specific assessments or recommendations that are then presented to clinicians for consideration. Study identification for the previous review3 involved searching the US National Library of Medicine MEDLINE electronic bibliographic database, Excerpta Medica (EMBASE), the International Information Service for the Physics and Engineering Communities (INSPEC), and the Science Citation Index (SCISEARCH), from 1974 to February 1992, for studies in any language. Conference proceedings and reference lists of relevant articles were also reviewed and authors were contacted. We used a similar strategy for the update, searching for any additional original articles on CDSSs from February 1992 to March 1998. The MEDLINE search strategy included the Medical Subject Heading (MeSH) terms computer-assisted decision making, artificial intelligence, computer-assisted diagnosis, computer-assisted therapy, and hospital information systems. The complete MEDLINE, EMBASE, and INSPEC search strategies are available on request. We searched SCISEARCH for references to the primary studies from the previous reviews and also searched the Cochrane Library6 (search strategy available on request) for potentially relevant citations. Reference lists from all relevant articles were examined and authors of relevant studies were contacted and asked if they were aware of any additional published or unpublished studies that we had not identified.
All citations, and index terms and abstracts if available, were reviewed and rated as "potentially relevant" or "not relevant." To assess interrater reliability, a random subset of 120 citations from the entire citation list was reviewed in duplicate. The raw agreement was 83% and the level of agreement beyond chance was 58% (κ=58%; 95% confidence interval [CI], 42%-75%).
Full-text articles were retrieved for all titles considered to be "potentially relevant" by either reviewer. All of these articles were reviewed independently in duplicate, and studies were included in the review if (1) the study participants were health professionals in clinical practice or postgraduate training; (2) the intervention was a CDSS evaluated in a clinical setting; (3) the outcomes that were assessed included clinician performance (a measure of the process of care) or patient outcomes (including any aspect of patient well-being); and (4) the study prospectively collected data, with a contemporaneous control group so that patient care with a CDSS was compared with care without one. The agreement between reviewers on the eligibility of studies for the systematic review was 98% and the level of agreement beyond chance was 86% (κ=86%; 95% CI, 76%-97%). All disagreements were resolved by consensus.
Study Evaluation
At least 2 authors assessed all selected studies independently for methodological quality. A 10-point rating scale was used and disagreements were resolved by consensus. The scale assessed 5 potential sources of bias, including the method of allocation to study groups (random vs quasi-random vs selected concurrent controls), the unit of allocation (clinic vs physician vs patient), the presence of baseline differences between groups that were potentially linked to the study outcome (no baseline differences or appropriate statistical adjustments made for differences vs baseline differences present and no statistical adjustment made vs unable to assess), the type of outcome measure (objective outcome or subjective outcome with blinded assessment vs subjective outcome with no blinding but clearly defined and explicit criteria for each outcome vs subjective outcome with no blinding of assessors and no explicit criteria for each outcome), and completeness of follow-up (>90% vs 80%-90% vs <80%).
The unit of allocation was included because of the possibility of contamination in trials in which interventions are applied to clinicians and end points are measured for patients. For example, the lowest score for unit of allocation was given when individual patients were allocated to intervention and control groups because a clinician may treat some patients with and others without the aid of the CDSS. Knowledge gained by the clinician from the CDSS may then be applied to control patients, leading to an underestimate of the system's effect. A similar situation can occur when individual clinicians within a team or group are the units of allocation: the presence of a CDSS or of colleagues who are using a CDSS in the clinical setting may influence the treatment given by clinicians allocated to the comparison group. While there are situations in which randomizing patients or physicians may be preferable to randomizing clinics,7 including cases where clinics differ in many respects and access to a large number of practices is limited, we chose to rate articles using the same scoring system as in the initial review3 for the sake of consistency.
All studies were initially graded for adequacy of follow-up of patients. In some trials, however, only a subset of the randomized patients were assessed for outcomes, for example, only those attending an appointment during the study period as a subset of all those randomized from a patient roster. This was especially common in trials assessing the role of CDSSs in preventive care. For these studies, the completeness of follow-up criterion was inapplicable and the remaining score was prorated so that the maximum possible score remained as 10 points.
A total of 40 new studies were assessed for scientific merit. The level of chance-corrected agreement between reviewers was high with a quadratic-weighted κ statistic of 82% (95% CI, 67%-97%). Disagreements were resolved by discussion.
Data Extraction and Analysis
One of the authors extracted information concerning patients, setting, intervention, and outcomes for each of the studies and this was verified by a second author. Subsequently, all studies that evaluated the effects of using a CDSS on patient outcomes and reported no statistically significant improvement were analyzed to determine the power of each to detect a moderate and clinically important difference in outcomes between the groups.8 An example of a moderate and clinically significant change would be a 25% absolute increase in the proportion of patients at a general medicine clinic who had adequate blood pressure control. The approach described by Cohen8 was used and in each case an α probability of .05 along with a medium effect size were used for the calculations.
The studies were then grouped into 4 major topic areas: drug dosing, diagnosis, preventive care, and studies involving other aspects of medical care. Because the studies differed in many respects, including clinical problems, clinicians, patients, method of intervention, follow-up, and measures of outcomes, no common measure of effect was considered to be justified for meta-analysis of the trial results. Thus, measures of the process of care and clinical outcomes were simply characterized for each study according to whether a statistically significant effect was reported.
RESULTS
In addition to the 28 studies previously reviewed,9-39 we identified 40 new studies that met our selection criteria and included sufficient results to be able to determine whether using the CDSS had affected health care practitioner behavior or patient outcomes.40-79 Several additional trials met our inclusion criteria but included insufficient results to determine the effect of using the CDSS.80-84
We also identified 9 review articles that discussed decision support systems. Four of these were narrative, nonsystematic reviews, including 2 that discussed the role of decision support in health care in general1, 85 and 2 that focused on specific aspects of decision support, including diagnostic aid systems86 and cancer prevention systems.87 The remaining 5 articles were systematic reviews. These included 2 prior publications by our group3, 5 and articles by Austin et al,2 Balas et al,88 and Shea et al.89 Austin et al completed a meta-analysis in 1994 of trials that assessed the effects of computer-based reminder systems on cervical cancer screening and tetanus immunization. This demonstrated a significant beneficial impact for both of these maneuvers. Balas and colleagues88 systematically reviewed the evidence from randomized controlled trials assessing numerous clinical information systems. They also reviewed the area of patient education. Their review demonstrated that reminder systems and drug dosing systems can improve health care practitioner performance. The meta-analysis by Shea and colleagues89 evaluated computer-based reminder systems for preventive care. They found benefits for vaccinations, breast cancer screening, colorectal cancer screening, and cardiovascular risk reduction but not for cervical cancer screening.
Linear regression of the validity scores of all trials in our review against their year of publication showed that study quality increased significantly over time (β=.14; 95% CI, 0.07-0.20). For the 28 studies that were identified in the original review, the scores for scientific merit ranged from 2 to 9 (mean, 6.4), with 7 (25%) of the studies having high scores between 8 and 10. The validity scores for the 40 new studies ranged between 4 to 10 (mean, 7.7), with 21 of the studies (53%) scoring 8 to 10. The majority of the trials were randomized. Only 9 studies (13%) used quasi-random allocation14, 16, 50, 63 or selected concurrent controls.24, 39-40,61, 69
Of the total of 68 included trials, 15 (22%) tested CDSSs designed to assist with dosing estimations for potentially toxic drugs (Table 1), 5 (7.0%) evaluated diagnostic aids, 19 (28%) assessed systems for preventive care (Table 2), and 29 (43%) tested programs for other aspects of medical care (Table 3). Almost all of these studies evaluated effects on clinician performance, but only 14 assessed patient outcomes. Of the 65 studies that evaluated the effect of using a CDSS on clinician behavior, 43 (66%) found at least some benefit. These included 9 (60%) of 15 studies on drug dosing systems, 1 (20%) of 5 studies evaluating diagnostic aids, 14 (74%) of 19 preventive care systems, and 19 (73%) of 26 studies assessing CDSSs for other medical care. Only 6 (43%) of the 14 studies that evaluated the effects on patient outcomes documented a benefit. However, 5 (62%) of the 8 trials that found no benefit for patient outcomes had a power of less than 80% to detect a moderate and clinically important improvement.
Table 1.—Studies of Computerized Drug Dosing*
Table 2.— Studies of Computerized Reminders for Preventive Care*
Table 3.—Studies of Other Aspects of Medical Care*
Drug Dosing Studies
Fifteen studies assessed systems designed to assist with drug dosing (Table 1), with 11 of these coming from the update. Eight studies9-10,41, 43-46,50 addressed the dosing of intravenous medications and 6 found improvements with the use of a CDSS. The systems that were tested were designed to assist with achieving or maintaining therapeutic theophylline or lidocaine hydrochloride levels or achieving improved anticoagulation control with heparin. Four of these trials also evaluated patient outcomes and only 1 found a significant benefit compared with usual clinical practice.46
The remaining 7 studies11-12,40, 42, 47-49 evaluated the role of CDSSs in warfarin dosing using a number of different end points. The time to reach a therapeutic international normalized rate (INR) or prothrombin time (PT) when initiating warfarin therapy was assessed by 3 trials. The findings were inconsistent. One study found no effect.48 In another study, the CDSS group reached therapeutic levels faster,11 while the reverse occurred in another trial.40 Using a CDSS decreased the time required to reach a stable INR or PT (in contrast to the first occurrence of a therapeutic INR or PT) in one trial,48 but had no effect in another.42 For maintenance therapy for outpatients, 1 study found that a CDSS was able to keep the INR within the therapeutic range more consistently than if a CDSS was not used,48 but 3 other studies found that using a CDSS did not improve the anticoagulation control.12, 47, 49 However, 1 of these 3 studies49 compared warfarin sodium dosing by a trainee physician with dosing by a nurse practitioner using a CDSS. Anticoagulation was equivalent in the 2 groups.
Diagnosis
Five studies13-17 assessed the role of CDSSs in diagnosis, all from the initial review. These evaluated a variety of systems designed to assist in the management of pediatric patients or patients presenting with chest or abdominal pain. In addition, Chase and colleagues13 tested a system that identified patients at high risk of respiratory tract complications postoperatively so that physiotherapy could be selectively provided. This was the only study to find a benefit with the CDSS. Positive effects were noted for both timely referral of patients for physiotherapy and reduced risk of postoperative complications.
Prevention
With the update, the number of studies of CDSSs providing preventive care reminders increased from 6 to 19 (Table 2). All of the studies evaluated clinician performance and 14 (74%) found a benefit for at least one of the processes of care measured. The majority of the systems in the original review reminded clinicians about a single test, such as blood pressure assessment,18, 21 vaccinations,19, 22 and Papanicolaou tests.20 In contrast, most of the systems tested in the new studies reminded physicians about several preventive care tests, including vaccinations and cancer screening. Also, the only study to look at prevention reminders for inpatients receiving acute medical care59 found no increase in the implementation of 22 preventive procedures, even though the majority of participating physicians indicated that they felt that acute hospitalizations provided an appropriate opportunity for reviewing and implementing preventive care.
Other Medical Care
Twenty-nine studies, including 16 new trials, assessed the role of CDSSs in other medical care (Table 3). Some of the systems that were tested were disease specific, focusing on hypertension, diabetes, or management of the acquired immunodeficiency syndrome, while others addressed numerous general medical problems (Table 3). Several systems were designed to improve drug prescribing and test ordering and others to decrease the number of admissions to hospital, to hasten hospital discharge, or to improve the functioning of anticoagulation clinics. Systems designed specifically to assist nurses have also been evaluated.
Almost all of these studies assessed the effects of using a CDSS on health care practitioner performance, and 19 (73%) of 26 studies found a benefit. The process of care for hypertension improved in one trial,24 but not in another.27 For diabetes care, compliance with recommendations improved in 2 of 4 studies.28, 68-70 The systems that provided patient-specific information about laboratory test results37, 64 found improvements. Compliance with recommendations for general medical problems also improved in the 5 studies29-31,71, 78 that addressed this. Safran and colleagues76 found that health care practitioners caring for patients with acquired immunodeficiency syndrome completed tasks more quickly when they received computer-based support. Fihn and colleagues66 noted that a CDSS designed to recommend follow-up periods for outpatients receiving warfarin anticoagulation at a thromboembolism clinic led to significantly longer periods between appointments, without any differences in the level of anticoagulation control between the 2 groups. No improvements were noted in the trials that evaluated drug prescribing or hospital admission practices. Using a CDSS did, however, lead to slightly earlier discharge from the hospital.77 Recently, Dexter and colleagues65 demonstrated that computer-generated reminders are effective at increasing the rate of advance directive discussions and form completion.
A smaller number of trials evaluated patient outcomes as end points; 4 (57%) of 7 found a benefit. Of the studies that assessed blood pressure, no improvements were noted for this end point.25, 27, 34-36 In contrast, beneficial changes in weight and patient quality of life were found in 2 studies.34-36,74 Also, fewer people who had been cared for with the assistance of a CDSS for the management of general medical problems subsequently required hospitalization or a chest or respiratory tract assessment.
COMMENT
This update documents considerable progress in the quantity and quality of studies of CDSSs. The number of high-quality, controlled trials has more than doubled since the previous review of studies up to 1992, and stronger conclusions can now be reached. For example, with 19 studies assessing preventive care systems in comparison with the limited set of 6 studies that were previously available and with 74% reporting positive findings, ambulatory care services and clinics should consider opportunities to acquire preventive care reminder systems—a conclusion reached in another recent review.88
Information relating to dosing systems has also increased since the initial review. Six studies, instead of 2, now indicate that potentially toxic, intravenously administered medications can be titrated more effectively using a CDSS than without one if the pharmacokinetics are well understood. Only 2 published studies found no improvement with a CDSS. While larger, confirmatory studies may be warranted (as concluded after the initial review), the consistent results suggest that it would now be reasonable to consider using a CDSS for this purpose in certain circumstances. The same cannot be said for warfarin dosing systems. Despite the fact that 7 studies have now addressed this topic, in comparison with the previous 2, the findings have remained inconsistent.
Perhaps not surprisingly, given the disappointing findings of earlier trials, there have not been any new controlled trials of diagnostic aid systems. Thus, our conclusions regarding these systems remain unchanged. For the time being they should only be deployed in settings in which they are being properly evaluated.
Also, given the rapid rate of development of CDSSs, the inconsistencies in results within classes of CDSSs, and the limited range of clinical settings in which CDSSs have been tested to date, it is important for health care centers to include some form of in-house evaluation when incorporating CDSSs. This is especially important if the system has not been previously tested, if the clinical setting differs substantially from that of prior testing sites, or if specially trained staff were included in the previous trials. The study of Overhage and colleagues59 is enlightening in this respect: despite the success of reminder systems for preventive care in ambulatory settings and the prior belief of clinicians that preventive care reminders would be welcomed for seriously ill, hospitalized patients, no benefits were detected when the reminders were implemented. Thus, as a minimum, adherence rates should be assessed after the implementation of any reminder system and compared with the preimplementation rates. Similarly, systems designed to alter laboratory testing behavior or adherence to disease management recommendations should have postimplementation evaluations to assess their impact.
To improve the chance that a decision support system will have a positive influence, the findings from 2 additional studies are worth considering. Litzelman et al90 demonstrated that requiring physicians to acknowledge reminders improved their adherence to recommendations. Recently, Lobach91 found that providing personalized feedback to physicians was an effective means of improving adherence to recommendations after the implementation of a CDSS.
Few studies of CDSSs to date have assessed patient outcomes and only a small proportion of these have found benefits. However, many of these studies have been too small to detect clinically important improvements in patient outcomes. It could be argued that this is mainly a matter of appropriate timing. Without more substantive effects on process, looking for clinical and economic benefits is premature. The CDSSs are improving so larger trials will be warranted soon. Given the expense and difficulties of doing such trials, this position has merit as long as the final stages of validation are not neglected—once more compelling process effects are shown. Unfortunately, no regulatory or other guarantee exists that such trials will be done, and market forces may drive deployment of such systems before adequate evaluation. Nevertheless, the increasing number and quality of CDSS trials in the past few years and the rapid assimilation of technological information into clinical settings bode well for the future of improving the effectiveness and efficiency of clinical care.
Acknowledgments
This research was supported by the Health Evidence Application and Linkage Network (HEALNet, a Canadian Network of Centres of Excellence Program). Dr Hunt is supported by a HEALNet fellowship. Dr Haynes is supported in part by a National Health Scientist award from the National Health Research and Development Program, Health, Ottawa, Ontario.
We thank the following experts who responded to our requests for additional information: Daniel Baer, Robert C. Burack, MD, PhD, Michael E. Burton, PharmD, Barry L. Carter, PharmD, D. Ezra, Stephen D. Fihn, MD, David A. Fitzmaurice, Joseph W. Hales, MD, Francis Lau, PhD, Daniel F. Lobach, PhD, MD, Clement J. McDonald, MD, Stephen J. McPhee, MD, Douglas K. Owens, MD, MSc, Michael Pringle, FRCGP, David M. Rind, MD, Arnika S. Ryff-de-Leche, Steven Shea, MD, Jonathon Teich, MD, William M. Tierney, MD, Barbara J. Turner, MD, MSc, Jeremy C. Wyatt, MD, and Rita Zielstorff, PhD. We are also grateful to Tom Flemming, MA, MLS, and Ann McKibbon, MLS, for expert assistance with the computerized searches.
Reprints: R. Brian Haynes, MD, PhD, Room 3H7, McMaster University Medical Center, 1200 Main St W, Hamilton, Ontario, Canada L8N 3Z5 (e-mail: bhaynes{at}fhs.mcmaster.ca).
References
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73.
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80.
- 81.
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
