 |
|
Interpreting Cost Analyses of Clinical Interventions
E. Andrew Balas, MD, PhD;
Rainer A. C. Kretschmer, MD;
Wolfgang Gnann, MBS;
David A. West, PhD;
Suzanne Austin Boren, MHA;
Robert M. Centor, MD;
Michael Nerlich, MD;
Mahendra Gupta, PhD;
Timothy D. West, PhD;
Naomi S. Soderstrom, PhD
JAMA. 1998;279:54-57.
ABSTRACT
| |
Objective.— In the present era of cost containment, physicians need reliable data about specific interventions. The objectives of this study were to assist practitioners in interpretation of economic analyses and estimation of their own costs of implementing recommended interventions.
Data Sources.— MEDLINE search from 1966 through 1995 using the text words cost or expense and medical subject heading (MeSH) terms costs and cost analysis, cost control, cost of illness, cost savings, or cost-benefit analysis.
Study Selection.— The 4 eligibility criteria were clinical trial with random assignment; health care quality improvement intervention tested; effects measured on the process or outcome of care; and cost calculation mentioned in the report.
Data Extraction.— After independent abstraction and after consensus development, financial data were entered into a costing protocol to determine which costs related to the intervention were provided.
Data Synthesis.— Of 181 articles, 97 (53.6%) included actual numbers on the costs of the intervention. Of 97 articles analyzed, the most frequently reported cost figures were in the category of operating expenses (direct cost, 61.9%; labor, 42.3%; and supplies, 32.0%). General overhead was not presented in 91 (93.8%) of the 97 studies. Only 14 (14.4%) of the 97 studies mentioned start-up costs. The text word $ in the abstract and the most useful MeSH index term of cost-benefit analysis appeared with nearly equal frequency in the articles that included actual cost data (37.1% vs 35.1%). Two thirds of articles indexed with the MeSH term cost control did not include cost figures.
Conclusions.— Statements regarding cost without substantiating data are made habitually in reports of clinical trials. In clinical trial reports presenting data on expenditures, start-up costs and general overhead are frequently disregarded. Practitioners can detect missing information by placing cost data in a standardized protocol. The costing protocol of this study can help bridge care delivery and economic analyses.
INTRODUCTION
REGARDLESS of the system of health care, there is an obvious and universal need for reliable information on the costs of clinical interventions. Physicians control more than 70% of health care expenditures.1 Thus, there is a need for physicians to consider costs in making patient care decisions.2 Medical textbooks highlight the need for individual physicians to know the cost of medicines they prescribe and to monitor the cost and effectiveness of clinical procedures. Since pharmacoeconomic statements play an important role in informing the public and providers, the Food and Drug Administration has become interested in more rigorous standards for cost data.3
Economic analyses currently available are often incomplete and fail to provide relevant cost information to practitioners. A recent series of position papers recommended the use of cost-effectiveness analyses to evaluate the underlying economics of clinical procedures and to identify those providing maximum health care benefits at the lowest costs.4-6 Those who perform cost-benefit, cost-effectiveness, and other economic analyses of health care may assume accurate data on expenses. However, assessment of efficiency is confused by diverse and conflicting interpretations of the term cost (eg, family expenditures,7 charges,8 direct costs,9 and productivity loss10). Approximately 25% of cost studies consider the costs of adverse effects or morbidity, or the savings accrued by averting costs through use of the intervention being tested.11
In addition to providing evidence regarding the outcome effect, randomized controlled clinical trials can serve as sources of reliable information on costs.12 The randomized controlled trial has achieved widespread recognition as the scientific "gold standard" for evaluating therapeutic efficacy.13 Trials that identify ineffective diagnostic and therapeutic interventions represent ways of controlling increasing costs of medical care. Beyond this role, specific cost calculations could help practitioners in identifying the most efficient health care technologies. Unfortunately, over the past 2 decades, the portion of trials that included an economic component has stayed relatively constant and low.14
The exploration of valuable pieces of cost information in the clinical literature offers more advantages to practitioners than rejecting all studies by focusing only on their shortcomings. The first objective of this study was to supplement expert opinion with empirical data regarding completeness, consistency, and comparability of intervention costs in reports on clinical trials. The second objective was to provide a benchmark template of cost categories to assist in interpreting cost calculations accompanying controlled testing of clinical interventions, and to assist practitioners in estimating their own costs as they try to implement recommendations.
METHODS
The study was conducted in 5 phases: execution of a systematic search strategy; screening for eligibility; information abstraction; consensus development; and analysis based on a structured costing protocol.
Search Strategy
The eligibility criteria for trials in this review were prospective, contemporaneously controlled clinical trial with random assignment; testing of health care quality improvement, a medical management intervention; measurement of effect on process or outcome of care; and calculation of cost of the intervention.
During phase 1, systematic MEDLINE searches were conducted. The title, abstract, and medical subject heading (MeSH) fields were searched for text words cost or expense and MeSH terms costs and cost analysis, cost control, cost of illness, cost savings, or cost-benefit analysis.Cost-effectiveness and cost-utility are not MeSH terms, but such studies can be identified by the character string of cost . The range of dates searched on MEDLINE was 1966 through 1995. Articles that met the first 3 eligibility criteria became part of the Columbia Registry, a systematic collection of randomized trial reports at the University of Missouri School of Medicine.15-18 Articles that met all 4 eligibility criteria were entered into phase 2.
During phase 2, trials retrieved through the search strategy were screened for a dollar sign ($) or any other symbol of currency. Those with a currency symbol were reviewed further. The inclusion criteria required that, in addition to retrieval by 1 of the previously listed text words or MeSH terms, there should be at least 1 cost figure presented that referred to cost of the intervention.
Abstraction
During phase 3, the remaining articles were reviewed and abstracted independently by 2 teams. Each team consisted of 2 persons who reviewed articles together. Qualifications of the reviewers included education in medicine and health services research. The 2 teams worked independently to check eligibility and abstract information. The same cost studies worksheet was used by the teams to collect information on the intervention, cost items, alternative units used in the trial, and reference to standardized methods. Only intervention-related expense data were collected. Monitoring and other research-related expenses appear in both the intervention and control groups; thus, such data were not collected.
During phase 4, there was consensus development on the abstraction of articles. If the teams did not agree on article inclusion or exclusion, or identification of cost items, the final decision was made independently by another person. Cohen  was calculated for comparison of the decisions to include or not include the article and the cost items identified by the 2 teams.19
The mean (SD) Cohen for the 2 teams on admitting a trial to the study was 0.660 (0.072). A rate of agreement above 0.6 is considered to be substantial.20 Initial judgment of reviewers was different for various cost items that were presented in the studies. It was often difficult to differentiate between costs of intervention and costs of consequences. Frequently, authors presented costs of intervention and costs of consequences in 1 table. Overall, the mean (SD) Cohen for the 2 teams on agreement regarding cost items was 0.859 (0.016), a rate of agreement considered to be excellent.21
Costing Protocol
Analysis of expenses arising from introduction and administration of a clinical intervention requires a replicable and generalizable costing protocol. The cost of the intervention is primarily the cost of resources and this study organized resources into specific categories.22 Analyzed costs reflected the expense of implementing the intervention only and did not include clinical trial expenses such as randomization and monitoring. In this study, based on well-defined and widely accepted activity-based accounting standards, cost items were placed in the following categories.23-25
Start-up costs were defined as those costs required for 1-time development of the intervention and included labor or personnel (salary and benefits required for human resources, subdivided into development or programming and hiring or training); new equipment or space (office and medical equipment or space needed to set up the intervention); consulting services (reimbursement for information on development of the intervention); and other costs.
Operating expenses were costs associated with operation of the intervention and included direct cost of intervention and general overhead. The direct cost of intervention category consisted of labor, supplies, communication expenses, and other costs. Labor costs included salary and benefits for clinical personnel and nonclinical support staff. Supply costs included costs of all medical and nonmedical supplies. Costs were not restricted to purchasing price of supplies, but may also have included cost of effort required for purchasing supplies and making them available. The general overhead category was divided into administration and occupancy. Occupancy was defined as rent, utilities, and telephone.
During phase 5, cost items were placed in the costing protocol. If there was doubt about composition of the cost figure presented, or if several different cost items were included in 1 cost figure, then the cost item was categorized higher on the hierarchy. For example, the cost of specific rooms was considered in some articles (eg, intensive care unit room or emergency department).26-27 However, it was not possible to determine whether the costs represented those for the space only, or also included costs for staff and equipment. Thus, these items were placed in the direct cost category of the intervention. Travel or transportation costs were considered as operating expenses and were categorized as direct cost in the subcategory of other.28 Office space was always categorized as general overhead in the subcategory of occupancy.
The completeness of cost calculation was measured by the number of cost items reported. Change over time was analyzed by calculating correlation between year of publication and completeness of cost calculation.
RESULTS
The systematic searches of this study resulted in retrieval of 181 trial reports. Ninety (49.7%) of the 181 articles have been published since 1991. The collected articles were published in 69 refereed and MEDLINE-indexed medical journals. All eligible studies presented cost data, but, as reflected by the abstract, cost-benefit analysis was the focus of only 4 studies, and cost-effectiveness analysis was the focus of 18 studies.
Table 1 provides information on the relationship between indexing and costing of interventions. One hundred one articles (55.8%) of the 181 articles were indexed by at least 1 of the 5 MeSH terms (costs and cost analysis, cost-benefit analysis, cost control, cost of illness, and cost savings).Cost-benefit analysis was the most useful MeSH term. However, 22.7% of the articles identified by the MeSH term cost-benefit analysis did not present cost data. The MeSH terms cost control, cost of illness, and cost savings were of little value in identifying articles for this study. More than 40% of articles in each of the 3 MeSH term categories, cost control, cost of illness, and cost savings, did not contain data on cost.
|
|
|
|
Table 1.—Relationship Between Indexing and Costing of Intervention for 181 Trials From the Columbia Registry
|
|
|
In phase 2, 84 of the 181 reports were excluded from detailed analysis for not having cost data. At first, 52 reports were excluded because cost data were considered to be trivial, or only costs of consequences were presented (eg, for an educational intervention only, yearly hospital costs, or physician charges were presented). The remaining 129 articles (71.3%) were further reviewed and abstracted independently by 2 teams. Thirty-two articles (17.7%) were additionally excluded for not specifically presenting costs of the intervention. Ultimately, 84 trials were excluded and only 97 trials (53.6%) were eligible for complete analysis.
Table 2 displays the categorization of the cost items as abstracted from the 97 trials eligible for this study. The categories presented in Table 2 are not mutually exclusive. Data from studies with more complete cost calculations appear in several hierarchical levels of cost categories. Fourteen articles (14.4%) provided start-up costs of the intervention. Sixty-three articles (65.0%) provided intervention operating expenses. Cost of labor was accounted for in 41 articles (42.3%) and cost of supplies was reported in 31 articles (32.0%). Sixty-seven articles (69.1%) provided the total cost of the intervention. Calculation of correlation between year of publication and completeness of cost calculation indicated lack of improvement over time (R, 0.01).
|
|
|
|
Table 2.—Appearance of Cost Items in Various Reported Calculations*
|
|
|
Table 3 presents examples of expression of costs in the 97 articles. More than 20 different expressions of cost were identified. These expressions were placed in 6 categories. Each cost may be expressed in different time units (eg, per day, month, or year). Some reports express costs in more than 1 unit of measurement. Therefore, categories in Table 3 are not mutually exclusive. The way these measures are expressed constitute the link between Table 2 and Table 3.
|
|
|
|
Table 3.—Ways of Expressing Costs in the Analyzed Literature*
|
|
|
Table 4 exemplifies the practical use of the costing protocol used in this study and shows how practitioners can place results of a study into the structured costing protocol and identify available and ignored costs. Data were abstracted from a trial report on cardiac rehabilitation soon after myocardial infarction.29 The selected trial report did comparatively well on several aspects (eg, cost data on new equipment, labor, and overhead). This example illustrates the point that start-up costs and general overhead categories, ignored by many studies, can account for more than 50% of all expenditures.
|
|
|
|
Table 4.—Application of Costing Protocol to Cardiac Rehabilitation Soon After Acute Myocardial Infarction*
|
|
|
COMMENT
The results of this study reflect variation in cost calculation and input for intervention costs, a fundamental part of medical economic analyses. It is unfortunate that statements regarding cost of care may be made without presenting data. Often-ignored cost items, like general overhead and start-up costs, also introduce a risk of unpleasant surprise at the end of implementation. Limitations of available clinical studies should be taken as a warning by practitioners and administrators who routinely make inferences and draw conclusions based on incomplete information. The costing protocol, described in the "Methods" section and illustrated in Table 4, shows the practical steps needed to collect direct cost data for economic analyses and to make pragmatic decisions regarding introduction of clinical interventions into patient care.
This data-driven exploration of actual costing practices and their shortcomings supplements available expert recommendations regarding costing. The widely accepted concepts of continuous quality improvement and evidence-based medicine stress the need for measurement and data instead of anecdotes and opinions in identifying opportunities for improvement. This study focuses on accounting categories of economic elements defined by the Panel on Cost-Effectiveness in Health and Medicine.22 For example, this analysis of costing practices indicated variability in expression of intervention costs. Per patient/year expression, a rarely-used approach, could probably facilitate comparison with changes in health status.
Despite the many inconsistencies of available studies, the clinical trial design can provide not only evidence regarding benefits but also insight into relative cost of one intervention vs another. Fortunately, none of the analyzed reports confused the clinical trial budget with costs that occur in the study group but not in the control group. Trials in which cost measures are built into the study are likely to do a better job of estimating costs than trials where measuring cost is an afterthought. Despite heightened interest in costs, technical and reporting recommendations for randomized controlled clinical trials do not specify requirements for cost analyses.30-33 The lack of data and inconsistencies of method should prompt researchers to supplement clinical trial checklists with a costing protocol.34-35
Calculations of costs of clinical interventions require radically different and more systematic approaches to produce replicable and generalizable information on cost-saving health care technologies. Appropriate methods are likely to increase the accuracy of cost estimates in clinical studies. There are methods to separate time spent by study personnel in activities critical to the clinical portion of the intervention from time spent in research-related activities.36 Appropriately categorizing all costs represents a first step to improve quality and comparability of cost information included in future clinical studies. Cost calculation studies, efforts to synthesize results of such studies, and ultimately those who are interested in cost-effective health care delivery could benefit from results of more systematic cost calculations.
AUTHOR INFORMATION
This research was supported in part by grant LM05545 from the National Library of Medicine.
Reprints: E. Andrew Balas, MD, PhD, Department of Health Management and Informatics, University of Missouri School of Medicine, 324 Clark Hall, Columbia, MO 65211.
From the Schools of Medicine (Dr Balas and Ms Boren) and Business and Public Administration (Dr West), University of Missouri, Columbia; School of Medicine, University of Regensburg, Germany (Drs Kretschmer and Nerlich and Mr Gnann); School of Medicine, University of Alabama, Tuscaloosa (Dr Centor); School of Business, Washington University, St Louis, Mo (Dr Gupta); College of Business, Iowa State University, Ames (Dr West); and School of Business, University of Washington, Seattle (Dr Soderstrom).
REFERENCES
| |
1. Levit KR, Lazenby HC, Cowan CA, Letsch SW. National health expenditures, 1990. Health Care Financ Rev. 1991;13:29-54.
PUBMED
2. Eddy DM. Clinical decision making from theory to practice: what do we do about costs? JAMA. 1990;264:1161, 1165, 1169-1170.
FREE FULL TEXT
3. Neumann PJ, Zinner DE, Paltiel AD. The FDA and regulation of cost-effectiveness claims. Health Aff (Millwood). 1996;15(3):54-71.
4. Russell LB, Gold MR, Siegel JE, Daniels N, Weinstein MC. The role of cost-effectiveness analysis in health and medicine. JAMA. 1996;276:1172-1177.
FREE FULL TEXT
5. Weinstein MC, Siegel JE, Gold MR, Kamlet MS, Russell LB. Recommendations of the panel on cost-effectiveness in health and medicine. JAMA. 1996;276:1253-1258.
FREE FULL TEXT
6. Siegel JE, Weinstein MC, Russell LB, Gold MR. Recommendations for reporting cost-effectiveness analyses. JAMA. 1996;276:1339-1341.
FREE FULL TEXT
7. Krahn MD, Berka C, Langlois P, Detsky AS. Direct and indirect costs of asthma in Canada, 1990. CMAJ. 1996;154:821-831.
ABSTRACT
8. Stempel DA, Hedblom EC, Durcanin-Robbins JF, Sturm LL. Use of a pharmacy and medical claims database to document cost centers for 1993 annual asthma expenditures. Arch Fam Med. 1996;5:36-40.
FREE FULL TEXT
9. Dor A, Held PJ, Pauly MV. The Medicare cost of renal dialysis: evidence from statistical cost function. Med Care. 1992;30:879.
FULL TEXT
|
WEB OF SCIENCE
| PUBMED
10. Taitel MS, Kotses H, Bernstein IL, Bernstein DI, Creer TL. A self-management program for adult asthma, part 2: cost-benefit analysis. J Allergy Clin Immunol. 1995;95:672-676.
FULL TEXT
|
WEB OF SCIENCE
| PUBMED
11. Udvarhelyi IS, Colditz GA, Rai A, Epstein AM. Cost-effectiveness and cost-benefit analysis in the medical literature. Ann Intern Med. 1992;116:238-244.
FREE FULL TEXT
12. Chalmers I. The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Ann N Y Acad Sci. 1993;703:156-163.
WEB OF SCIENCE
| PUBMED
13. Robinson R. Economic evaluation and health care: what does it mean? BMJ. 1993;307:670-673.
FREE FULL TEXT
14. Strauss MJ, Bleecker GC, Steinwald AB. Cost-effectiveness analysis in a changing health care environment. Eur J Cancer. 1993;29A(suppl 7):S3-S5.
15. Spencer DC, Balas EA, Nashelsky JE. Information management. In: Taylor RB, ed. Family Medicine Principles and Practice. 4th ed. New York, NY: Springer-Verlag NY Inc; 1994:1028-1035.
16. Balas EA, Austin SM, Ewigman BG, Brown GD, Mitchell JA. Methods of randomized controlled clinical trials in health services research. Med Care. 1995;33:687-699.
WEB OF SCIENCE
| PUBMED
17. Balas EA, Stockham MG, Mitchell JA, Austin SM, West DA, Ewigman BG. The Columbia Registry of Information and Utilization Management Trials. J Am Med Inform Assoc. 1995;2:307-315.
FREE FULL TEXT
18. Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of computerized information services: a review of 98 randomized clinical trials. Arch Fam Med. 1996;5:271-278.
FREE FULL TEXT
19. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Measurement. 1960;20:37-46.
FULL TEXT
20. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.
FULL TEXT
|
WEB OF SCIENCE
| PUBMED
21. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New York, NY: John Wiley & Sons Inc; 1981:218.
22. Gold MR, ed, Siegel JE, ed, Russell LB, ed, Weinstein MC, ed. Cost-effectiveness in Health and Medicine. New York, NY: Oxford University Press; 1996.
23. Atkinson AA, Banker RD, Kaplan RS, Young MS. Management Accounting. Englewood Cliffs, NJ: Prentice-Hall International Inc; 1995.
24. Garrison RH, Noreen EW. Managerial Accounting. 8th ed. Chicago, Ill: Richard D Irwin Inc, a Times-Mirror Higher Education Group Inc Co; 1997:178-199.
25. Cooper R, Kaplan RS. Activity-based systems: measuring the costs of resource usage. Accounting Horizons. 1992;6:1-13.
26. Hughes SL, Cummings J, Weaver F, Manheim LM, Conrad KJ, Nash K. A randomized trial of Veterans Administration home care for severely disabled veterans. Med Care. 1990;28:135-145.
FULL TEXT
|
WEB OF SCIENCE
| PUBMED
27. Topol EJ, Burek K, O'Neill WW, et al. A randomized controlled trial of hospital discharge 3 days after myocardial infarction in the era of reperfusion. N Engl J Med. 1988;318:1083-1088.
ABSTRACT
28. Tucker MA, Davison JG, Ogle SJ. Day hospital rehabilitation—effectiveness and cost in the elderly: a randomised controlled trial. Br Med J (Clin Res Ed). 1984;289:1209-1212.
FREE FULL TEXT
29. Oldridge N, Furlong W, Feeny D, et al. Economic evaluation of cardiac rehabilitation soon after myocardial infarction. Am J Cardiol. 1993;72:154-161.
FULL TEXT
|
WEB OF SCIENCE
| PUBMED
30. Working Group on Recommendations for Reporting Clinical Trials in the Biomedical Literature. Call for comments on a proposal to improve reporting of clinical trials in the biomedical literature: a position paper. Ann Intern Med. 1994;121:894-895.
FREE FULL TEXT
31. The Standards of Reporting Trials Group. A proposal for structured reporting of randomized controlled trials. JAMA. 1994;272:1926-1931.
FREE FULL TEXT
32. Rennie D. Reporting randomized controlled trials: an experiment and a call for responses from readers. JAMA. 1995;273:1054-1055.
FREE FULL TEXT
33. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996;276:637-639.
FREE FULL TEXT
34. Mason J, Drummond M. Reporting guidelines for economic studies. Health Econ. 1995;4:85-94.
WEB OF SCIENCE
| PUBMED
35. Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the British Medical Journal: the BMJ Economic Evaluation Working Party. BMJ. 1996;313:275-283.
FREE FULL TEXT
36. Oddone E, Weinberger M, Hurder A, Henderson W, Simel D. Measuring activities in clinical trials using random work sampling: implications for cost-effectiveness analysis and measurement of the intervention. J Clin Epidemiol. 1995;48:1011-1018.
FULL TEXT
|
WEB OF SCIENCE
| PUBMED
 CiteULike  Connotea  Del.icio.us  Digg  Reddit  Technorati  Twitter
What's this?
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
|
Health Economics of Chronic Obstructive Pulmonary Disease
Halpin
Proc Am Thorac Soc 2006;3:227-233.
ABSTRACT
| FULL TEXT
The Seattle-King County Healthy Homes Project: A Randomized, Controlled Trial of a Community Health Worker Intervention to Decrease Exposure to Indoor Asthma Triggers
Krieger et al.
AJPH 2005;95:652-659.
ABSTRACT
| FULL TEXT
Cost-effectiveness of Combined Outreach for the Pneumococcal and Influenza Vaccines
Weaver et al.
Arch Intern Med 2001;161:111-120.
ABSTRACT
| FULL TEXT
Outcomes Research in Regional Anesthesia and Analgesia
Wu and Fleisher
Anesth. Analg. 2000;91:1232-1242.
FULL TEXT
The Quality of Reporting in Published Cost-Utility Analyses, 1976-1997
Neumann et al.
ANN INTERN MED 2000;132:964-972.
ABSTRACT
| FULL TEXT
Improving Preventive Care by Prompting Physicians
Balas et al.
Arch Intern Med 2000;160:301-308.
ABSTRACT
| FULL TEXT
Cost Utility Analysis of Radiographic Screening for an Orbital Foreign Body before MR Imaging
Seidenwurm et al.
Am. J. Neuroradiol. 2000;21:426-433.
ABSTRACT
| FULL TEXT
|
