Relationship Between Methodological Trial Quality and the Effects of Selective Digestive Decontamination on Pneumonia and Mortality in Critically Ill Patients
- Christianne A. van Nieuwenhoven, MD;
- Erik Buskens, MD, PhD;
- Frank H. van Tiel, MD, PhD;
- Marc J. M. Bonten, MD, PhD
- Author Affiliations: Department of Medical Microbiology, University Hospital Maastricht (Drs van Nieuwenhoven and van Tiel), Julius Center for General Practice and Patient Oriented Research (Dr Buskens) and Department of Internal Medicine, Division of Infectious Diseases and AIDS (Dr Bonten), University Hospital Heidelberglaan, Utrecht, the Netherlands.
Abstract
Context Although meta-analyses of randomized trials have shown that selective digestive decontamination (SDD) prevents nosocomial pneumonia in critically ill patients, the influence of trial quality on the effectiveness of SDD has not been rigorously evaluated.
Objective To assess the methodological quality of individual studies of SDD and its relation to the reported effects on pneumonia and mortality.
Design Thirty-two studies were identified in a MEDLINE and reference list search and their methodological quality was assessed using a scoring system (range, 0-13 points) based on allocation and concealment, patient selection, patient characteristics, blinding of the intervention, and the definition of pneumonia.
Main Outcome Measure Methodological quality of the primary trials and its effect on the relative risk reductions (RRRs) of SDD on pneumonia and mortality.
Results The mean (SD) methodological quality score was 7.8 (2.9) (range, 1-11). The RRRs ranged from –0.1 to 1.0 for pneumonia and from –0.1 to 0.6 for mortality. The methodological quality score was associated with the RRR for pneumonia so that for each quality-point added, the RRR decreased by 5.8% (95% confidence interval, 2.4%-9.3%). No association between trial quality and the impact of SDD was found on mortality. Of the individual trial quality characteristics, patient selection, allocation of intervention, and blinding most strongly influenced the treatment effect.
Conclusions The inverse relationship between methodological quality score and the benefit of SDD on the incidence of pneumonia may have resulted in overly optimistic estimates of SDD in prior meta-analyses. This emphasizes the importance of rigorous trial design in evaluating preventive interventions in the intensive care unit.
Nosocomial pneumonia is the most frequent infection among mechanically ventilated patients treated in intensive care units (ICUs) and has been associated with increased morbidity, antibiotic use, and prolonged length of stay. Prevention of nosocomial pneumonia, therefore, remains a challenge for intensive care medicine.1
Selective decontamination of the digestive tract (SDD) is the most extensively studied method of infection prevention for critically ill patients. The SDD method consists of eradicating potentially pathogenic microorganisms in the oral cavity by application of nonabsorbable antibiotic paste and decontamination of the rest of the gastrointestinal tract by local administration of the same antibiotics, conducting a short course of systemic prophylaxis for respiratory tract infections that may occur during the first days of intubation, regulary taking cultures of throat swabs and feces to monitor the effectiveness of SDD, and maintaining optimal hygiene to prevent cross-infection. More than 30 studies and 6 meta-analyses2-7 have been published. Although all meta-analyses showed significant reductions in the incidence of pneumonia among patients receiving SDD, the effects on mortality are less clear. A recent meta-analysis2 suggested an overall 20% reduction in ICU mortality. In another meta-analysis,3 a significant 40% reduction in ICU-mortality was found among surgical patients, but no significant difference was found for medical patients.3 Interestingly, reductions in mortality were found only for patients who had received systemic and topical antibiotic prophylaxis.2-3 Although some have argued that SDD is now an evidence-based intervention with impressive positive results for patient care,8 others question the value of meta-analyses as a method for hypothesis testing.9-11 However, the quantity of studies does not necessarily reflect quality of studies. An inadequate approach to controlled trial design and execution may be associated with bias in estimating treatment effects, and incorporating such studies in meta-analyses may bias the effect of interventions.12-13 In our study, we analyzed the relationship between the methodological quality of individual SDD studies and the reported effects on pneumonia and mortality.
METHODS
Assessment of Methodological Quality
We searched MEDLINE and reference lists of previous meta-analyses for relevant trials on SDD. We searched for randomized trials using the initialism SDD. Studies that were only published as abstracts were excluded because data were insufficient for a thorough quality assessment. Criteria to assess the methodological quality of individual trials were partly derived from a previously published scoring method for prevention of nosocomial pneumonia, which includes aspects of treatment allocation and concealment, patient selection, comparability of patient groups, blinding, and diagnosis of pneumonia.14
Zero, 1, or 2 points were given for each of 5 criteria. We slightly modified scoring criteria for diagnosing pneumonia by adding a point when pneumonia was diagnosed with quantitative cultures of endotracheal aspirates or samples obtained via bronchoscopic techniques. Several studies have demonstrated that adding these techniques to standard diagnostic criteria improves the specificity for diagnosing pneumonia, and their use has, therefore, been advised.15 Furthermore, randomization was separated into the method of allocation and the method of concealment (ie, how the assignment sequence is blinded and therefore protected against manipulation). As a result, the maximum score for study quality was 13 (Table 1). When analyzing the relationship between mortality and study quality, the category of diagnosing pneumonia was not included. Studies were considered high-quality studies when the total score was 7 points or more. Low-quality studies scored fewer than 7 points. A cutoff of 5 was used for this analysis on patient mortality (without points for diagnosing pneumonia).
Table 1. Criteria for Assessment of Methodological Quality
Since the quality scoring system gives equal weight to each characteristic summed in a total score, an additional analysis of individual quality characteristics was performed. For this analysis, components of trial quality were dichotomized: high quality for any characteristic was defined as the highest possible score of 2, and scores of 0 or 1 were considered low quality.
Data Analysis
Three researchers (C.A.v.N., F.H.v.T., and M.J.M.B) independently reviewed all studies. In cases of disagreement about any criteria, agreement was reached by consensus. The relative risk reductions (RRRs) with 95% confidence intervals (CIs) were calculated for pneumonia and mortality for each of the trials.16 Linear regression analysis was performed with methodological quality score (Table 1) as the independent variable; pneumonia and mortality were the dependent variables. Since SDD has been reported to be more efficacious in surgical than in medical patients,3 relationships between quality scores and proportion of surgical patients in each study were also analyzed using linear regression. Subsequently, we entered dichotomized characteristics in the univariable analysis to asses the influence of individual characteristics on outcomes. Characteristics with significant associations were then included in multivariate linear regression analysis. A P value of less than .05 was considered statistically significant.
RESULTS
Thirty-two trials were included (Table 2). In 1 study,31 2 SDD-schedules were compared with a single control group, and both analyses have been included separately. In another study in which SDD was compared with a simultaneously studied as well as a historical control group, only the comparison with the simultaneously studied control group was included.41
Table 2. Methodological Quality and Relative Risk Reductions (RRRs) for Pneumonia and Mortality*
Overall Quality Score
The mean (SD) methodological quality score was 7.8 (2.9) (range, 1-11). Reviewers reached agreement in 61% of the studies. They differed by 1 point in 31% of the studies. In the remaining studies, scores from 1 reviewer differed by 2 points (5%), or all 3 reviewers arrived at different scores (3%).
Relative risk reductions for pneumonia ranged from –0.1 (incidence higher in patients receiving SDD) to 1 (complete prevention of pneumonia). Significant reductions in the incidence of pneumonia were reported in 21 studies. When comparing all patients receiving SDD (n = 2400; incidence, 12%) with control patients (n = 2404; incidence, 29%), the RRR for pneumonia was 0.57 (95% CI, 0.49-0.65). Relative risk reductions for the 12 studies with high quality scores were 0.68 (95% CI, 0.57-0.95), and for the 18 studies with low quality scores 0.47 (95% CI, 0.36-0.58).
Relative risk reductions for mortality ranged from –0.15 to 0.59, but a significant decrease in mortality was not found in any individual trial. When all the studies were grouped, mortality rates were 25% for SDD patients and 28% for controls, resulting in an RRR of 0.12 (95% CI, 0.03-0.21). Relative risk reductions for mortality were − 0.14 (95% CI, − 1.14 to 0.87) for studies with low quality scores and 0.14 (95% CI, 0.03-0.25) for studies with high quality scores. Relative risk reduction for mortality was 0.18 (95% CI, 0.04-0.32) when only studies using systemic prophylaxis as part of SDD with quality scores of 5 or more were analyzed.
A significant association was found for methodological quality scores and RRRs for pneumonia (Figure 1). For each point added to the methodological quality score, the RRR decreased by 5.8% (95% CI, 2.4% to 9.3%). Interestingly, the variance in RRRs was largest for studies with high quality scores. No such association was found for RRRs of mortality. The calculated reduction in mortality per point reduction of trial quality was –1.7% (95% CI, –3.7% to 0.03%). When studies with and without systemic prophylaxis were analyzed separately, results remained unchanged (data not shown).
Figure. Relationship Between Methodological Quality Score and Reduction of Ventilator-Associated Pneumonia and Mortality
Quality Characteristics
Of the trial quality features we considered, patient selection, allocation of the intervention and blinding were significantly associated with the RRRs of pneumonia. Blinding remained significantly associated in multivariate analysis (Table 3). For mortality, none of the trial quality characteristics were significantly associated with outcome.
Table 3. Univariate and Multivariate Logistic Regression Analysis of Quality Characteristics for Developing Pneumonia and Mortality in Intensive Care Units*
COMMENT
Our analysis demonstrated an inverse relationship between the methodological quality of SDD studies and the observed effects on the incidence of nosocomial pneumonia: the higher the quality score, the smaller the RRR of pneumonia. No such association was found for mortality. The trial quality characteristics associated with the outcome of pneumonia were patient selection, allocation of the intervention, and blinding. When creating new methodological quality scores, or adapting currently existing scores for meta-analysis, the relative contributions of each characteristic should be taken into account.
Studies on the prevention of nosocomial pneumonia in ICU are methodologically challenging. Most importantly, to date, no criterion standard test exists for the primary end point: nosocomial pneumonia. Therefore, vigorous attempts to minimize bias through other means is required. For example, the many cointerventions that may influence the development of pneumonia should be described if not controlled to aid in trial interpretation; such cointerventions were rarely reported in this literature. Attempts to minimize bias through duplicate blinded outcome assessment would be desirable.
Large numbers of patients from a homogenous patient population should be included in prospective, randomized, placebo-controlled trials. Blinding of whom for what purpose should be reported. Most of these trials included a small number of patients leading to the possibility of type I and type II errors, which can be magnified in meta-analyses. Our finding of an inverse relationship between study quality and reported benefits on the incidence of pneumonia in patients receiving SDD strongly suggests that the quality of study design and execution has influenced reported benefits of SDD. The fact that study quality influenced the effectiveness of pneumonia prevention is supported by our findings that study quality did not influence the reported effects on mortality. Interestingly, we found a small but significant reduction in mortality when analyzing studies of high quality. No reduction in mortality was found in low-quality studies although the reduction in incidences of pneumonia was 68%. If pneumonia and ICU mortality are associated, similar trends between both end points and study quality were to be expected. The absence of this association could be explained by inaccurate estimates of the incidence of pneumonia, particularly in unblinded trials.
In addition to possible bias in the trials on the prevention of nosocomial pneumonia and difficulties in interpretation of meta-analyses, the use of a single quality score may also influence study outcome.19 Using 25 different quality scales, Juni et al48 demonstrated that different quality scoring scales may lead to different conclusions from meta-analyses. They concluded that relevant methodological aspects should be assessed separately and that their influences on effect sizes should be explored.48 We screened most of the quality scoring scales used by Juni et al,48 and almost all included variables such as patient selection, randomization, and blinding procedures but not specific criteria for diagnosing pneumonia. Therefore, we decided to use the scoring system designed for this patient population and this specific infection.14 When assessing the importance of different methodological variables separately, we found differences in their relative importance. This suggests that by simply adding up scores based on perceived relevance may not be appropriate and that informed weighting of quality scores according to methodological importance may be more suitable.
An association between study quality on the benefits of an intervention summarized in a meta-analysis, as reported herein, confirms the results of previous analyses in other research areas. Analyzing 250 controlled trials from 33 meta-analyses, Schulz et al12 determined that the exaggerated estimates of treatment effects were 30% to 41% for trials in which concealment was either unclear or inadequate and 17% for trials that were not double-blind. In addition, Moher et al13 reanalyzed 127 randomized controlled trials included in 11 meta-analyses and quantitatively determined study quality. Compared with high-quality trials, low-quality trials were associated with an increased estimate of benefit of 34%, and trials that used inadequate allocation concealment were associated with an increased estimate of benefit of 37% compared with trials using adequate methods.13
The routine use of SDD as a measure of infection prevention in the ICU remains controversial. Whether SDD significantly reduces ICU mortality would be most convincing if it were confirmed in a large, prospective, randomized, double-blind study. Moreover, beneficial effects on secondary end points, such as duration of mechanical ventilation and ICU stay, and amount of antibiotic use remain unproven, whereas selection of antibiotic-resistant pathogens remains a cause for concern and must be determined from longitudinal studies or modelled in a cost-effectiveness analysis. Unfortunately, the short time horizon of these SDD trials means we have insufficient information about long-term microbial consequences of SDD. Our findings have implications in practice and research. We have shown the importance of considering randomization trial quality in making appropriate inferences about using SDD for pneumonia prevention in the ICU. The results for this methodological analysis emphasize the need for rigorous methods to avoid biased randomized trials and also biased meta-analyses of these trials.
Acknowledgments
Author Contributions: Study concept and design: Buskens, Bonten.
Acquisition of data: van Nieuwenhoven, van Teil, Bonten.
Analysis and interpretation of data: Buskens, Bonten.
Drafting of the manuscript: Buskens, Bonten.
Critical revision of the manuscript for important intellectual content: van Nieuwenhoven, Buskens, van Teil, Bonten.
Statistical expertise: Buskens.
Administrative, technical, or material support: van Nieuwenhoven.
Study supervision: Buskens, Bonten.
Funding/Support: This work was funded by grant 28-2125-2 from the Praevention Foundation.
Acknowledgment: We thank Margriet Schneider for her comments.
Advisory Board: David Bihari, MD; Christian Brun-Buisson, MD; Timothy Evans, MD; John Heffner, MD; Norman Paradis, MD.
Corresponding Author and Reprints: Marc J. M. Bonten, MD, PhD, Department of Internal Medicine, Division of Infectious Diseases and AIDS, University Hospital, Heidelberglaan 100, 3584 CX Utrecht, the Netherlands (e-mail: m.bonten{at}wxs.nl).
