Safety Outcomes in Meta-analyses of Phase 2 vs Phase 3 Randomized Trials
Intracranial Hemorrhage in Trials of Bolus Thrombolytic Therapy
- John W. Eikelboom, MBBS, MSc, FRACP;
- Shamir R. Mehta, MD, FRCPC;
- Janice Pogue, MSc;
- Salim Yusuf, MBBS, DPhil, FRCPC
Abstract
Context Recent studies have reported disagreement between meta-analysis of small trials and subsequent large trials addressing the same question. However, disagreement for uncommon but serious adverse safety outcomes has not been examined.
Objective To explore disagreement for serious adverse safety (intracranial hemorrhage [ICH]) and efficacy outcomes between meta-analysis of phase 2 (small) vs meta-analysis of phase 3 (large) randomized controlled trials comparing the efficacy of bolus thrombolytic therapy with infusion for acute myocardial infarction (AMI).
Data Sources Electronic databases (MEDLINE, Cochrane Database of Clinical Trials) between January 1980 and December 1999 using the search terms thrombolysis, thrombolytic therapy, and myocardial infarction; conference proceedings; and reference lists.
Study Selection Fifteen randomized trials comparing thrombolytic agents administered by bolus injection with standard infusion therapy in patients with AMI.
Data Extraction Data on ICH, other causes of stroke, total mortality, and reinfarction were independently extracted from each study by 2 observers.
Data Synthesis Meta-analysis of 9 phase 2 trials (n = 3956) revealed a lower risk of ICH with bolus thrombolytic therapy (odds ratio [OR], 0.53; 95% confidence interval [CI], 0.27-1.01), which was not statistically significant. Meta-analysis of 6 phase 3 trials (n = 62 673) indicated a significant increase in risk of ICH (OR, 1.25; 95% CI, 1.06-1.49). These results were significantly different (P = .01). There was no disagreement for efficacy outcomes. Phase 2 trials included younger and heavier patients with lower baseline blood pressures, and were more often open-label. Subgroup analyses suggested that each of these factors was associated with a lower estimate of risk of ICH with bolus agents.
Conclusions Our results suggest that when therapeutic interventions are associated with a potential for uncommon but serious adverse safety outcomes, there may be differences between small phase 2 and large phase 3 trials that result in their disagreement for safety but not necessarily efficacy outcomes. Further investigation of the frequency and causes of disagreement between small and large trials for safety outcomes is warranted.
- KEYWORDS:
- adverse reaction
- intracranial hemorrhages
- meta-analysis
- randomized trials
- thrombolytic therapy
Promising results seen in meta-analyses of small randomized trials may no longer be evident when more definitive large trials are performed.1-5 One potential cause of disagreement not previously considered relates to the situation when the therapeutic intervention under evaluation is known to be associated with the potential for a serious adverse safety outcome. Because serious adverse safety outcomes are uncommon, small phase 2 trials are almost always underpowered to reliably detect significant numbers of these events.6 Nevertheless, one of the primary roles of phase 2 trials is to provide essential safety information about new therapies.6-7 Since phase 2 trials are also among the first clinical studies to evaluate the safety and efficacy of a new therapeutic intervention, they are generally very closely monitored and test a relatively small and narrowly defined patient population.8 To minimize the risk of premature termination of the development of an otherwise promising new drug or class of drugs, patients entered into phase 2 trials may be specifically selected to minimize the risk of a known serious adverse safety outcome.
By contrast, subsequent large phase 3 trials are concerned primarily with demonstrating efficacy and, in order to enhance their generalizability, tend to include a broader range of patients. Therefore, when there is potential for serious adverse safety outcomes, there may be a greater potential for disagreement between phase 2 and phase 3 trials, and between meta-analyses of phase 2 and phase 3 trials.
Recent years have seen a burgeoning literature on the use of bolus thrombolytic therapy for the in-hospital treatment of acute myocardial infarction (AMI). The results of most phase 2 trials comparing newer bolus agents with infusion thrombolytic therapy have been highly promising, suggesting that bolus agents provide superior efficacy and a reduced risk of intracranial hemorrhage (ICH). However, a recent meta-analysis of the phase 3 trials demonstrated no efficacy advantage but an increase in ICH with bolus vs standard infusion thrombolytic therapy (odds ratio [OR], 1.25; 95% confidence interval [CI], 1.08-1.45).9
The objective of the present study was to critically appraise recent randomized trials of bolus vs infusion thrombolytic therapy for AMI and to examine possible disagreement between the phase 2 and the phase 3 trials for ICH and for efficacy outcomes including nonhemorrhagic stroke, reinfarction, and all-cause mortality.
METHODS
Data Sources and Study Selection
We sought to obtain data on ICH, stroke, AMI, and death from completed, published and unpublished, unconfounded randomized trials comparing bolus with standard infusion of thrombolytic therapy in AMI. Relevant studies were identified by manual searches of conference proceedings and reference lists, and by searching electronic databases (MEDLINE, Cochrane Database of Clinical Trials) from January 1980 to December 1999 using the terms thrombolysis, thrombolytic therapy, and myocardial infarction.
To be included, trials had to (1) be randomized; (2) include patients with AMI; (3) compare thrombolytic therapy administered by bolus injection (each bolus given over ≤5 minutes) with that administered by infusion (≥30 minutes); and (4) provide objective confirmation of the diagnosis of ICH based on the results of cranial computed tomography (CT) or magnetic resonance imaging (MRI) scan. We did not consider trials evaluating the use of bolus anistreplase since these trials were largely designed and/or conducted prior to 1990, when CT or MRI was not routinely used to confirm the diagnosis of ICH.
Definition of Phase 2 and Phase 3 Trials
Phase 2 trials were defined as pilot studies designed to assess the effectiveness and safety of a drug based on a surrogate outcome (eg, angiographic blood flow determined during coronary angiography). These were also generally small trials involving fewer than 1000 patients. Phase 3 trials were defined as those designed to evaluate the effectiveness and safety of a drug based on clinical outcomes.10 These were also generally larger trials, involving 1000 or more patients.
Study Quality
Criteria used to assess the study quality were concealment of randomization, blinding of the patient and investigator to the intervention (double-blinding), completeness of follow-up (>95%), and use of an intention-to-treat primary analysis. Data on ICH, other causes of stroke, total mortality, and reinfarction were independently extracted from each study by 2 observers.
Statistical Analysis
We pooled results for each outcome separately from phase 2 trials, phase 3 trials, and all trials combined. Our primary statistical approach was based on a modified Mantel-Haenszel method.11-12 Unlike a random-effects model, this approach is based on no assumptions except that the treatment effect can be expected to be in the same overall direction in different patient populations. Heterogeneity among the individual studies is not incorporated in the summary estimate. Results are presented as ORs and 95% CIs together with corresponding P values.
Publication bias was further explored with a funnel plot of effect size (OR of ICH) vs study precision.4 In the absence of publication bias, small and large studies should be symmetrically distributed about the summary estimate of the effect size. The effect size estimates from small, less precise studies (phase 2 trials) will be expected to be more widely spread than those from the large, more precise studies (phase 3 trials), so that data from all the studies should take the shape of a funnel. In the presence of bias against publication of small studies, the wide end of the funnel will be distorted or missing. The regression asymmetry test for publication bias devised by Egger et al4 and the rank-correlation test of Begg and Mazumdar13 were used to formally assess funnel plot asymmetry.
Subgroup Analyses
We explored the possible impact of differences in trial design on the pooled risk estimate for ICH by comparing phase 2 and phase 3 trials as well as subgroups of trials according to their baseline patient characteristics, fibrin-specificity of the bolus thrombolytic agent, and indicators of study quality. These variables were selected because they have been reported to be a cause of heterogeneity in meta-analysis of small vs large trials (blinded vs open-label trial, intention-to-treat vs efficacy analysis),2-3,5 or predictors for risk of ICH with thrombolytic therapy (age >75 years, lower body weight, higher systolic blood pressure, fibrin specificity of the bolus agent).14-15
We arbitrarily categorized trials according to whether their estimates of central tendency (mean or median) for age, body weight, and systolic blood pressure were less than, equal to, or greater than the overall mean for all the trials. Overall means for phase 2 trials, phase 3 trials, and for all the trials combined were calculated by averaging the means (or medians) for the individual trials after weighting them by study size (number of patients randomized). For discrete variables, trials were arbitrarily categorized according to ages of patients enrolled (≤75 years, or >75 years), was blinded vs open-label, used a primary intention-to-treat analysis vs an efficacy analysis, or evaluated a highly fibrin-specific bolus thrombolytic agent (tenecteplase or staphylokinase) vs a first- or second-generation bolus agent (saruplase, alteplase, reteplase, urokinase, lanoteplase, E6010). The OR for ICH was determined in different trial strata defined by these categories, and we looked for statistical evidence of heterogeneity across the strata.
RESULTS
Studies Included
Thirty-nine potentially eligible studies evaluating the use of bolus thrombolytic therapy were identified, of which 26 were considered for inclusion after initial screening of the abstracts.16-41 Fifteen trials involving a total of 66 686 patients met our inclusion criteria.16-30 Excluded trials did not use standard infusion thrombolytic therapy in the control arm (n = 11).31-41
Key features of study design were the number of patients randomized, age eligibility, therapeutic interventions, and the primary outcome (Table 1). The primary outcome in all phase 3 trials was all-cause mortality at 30 through 35 days, while the primary outcome in 8 of the 9 phase 2 trials was blood flow demonstrated angiographically. The remaining phase 2 trial28 reported the composite of death, myocardial infarction, or disabling stroke as the primary outcome, but was underpowered for this outcome and also reported blood flow demonstrated angiographically.
Table 1. Phase 2 and Phase 3 Randomized Trials of Bolus vs Infusion Thrombolytic Therapy in AMI*
Comparison of Phase 2 and Phase 3 Trials
Summary baseline characteristics, types of interventions, and quality indicators of phase 2 vs phase 3 trials are compared in Table 2. Compared with patients in phase 3 trials, those in phase 2 trials appeared to have a lower overall mean age and mean baseline systolic blood pressure, and a higher mean baseline body weight. Five phase 2 trials excluded patients older than 75 years but only 1 phase 3 trial24 had an upper age restriction, which was set at 80 years. In phase 2 trials, 34% of patients had a history of hypertension compared with 38% in the phase 3 trials. A highly fibrin-specific bolus thrombolytic agent was evaluated in 2 phase 2 trials and 1 phase 3 trial.
Table 2. Comparison of Phase 2 and Phase 3 Randomized Trials of Bolus vs Infusion Thrombolytic Therapy in AMI*
All trials in which the relevant quality data were reported appear to have been properly randomized (with adequate concealment),16-17,19-20,22, 27, 29 and had at least 95% follow-up.16-20,22-23,25, 28-29 Of phase 2 trials, 6 were open-label vs 2 phase 3 trials, and 2 phase 2 trials reported an intention-to-treat analysis vs 100% of phase 3 trials for which these data were available.
Safety Outcomes: ICH
Figure 1 and Figure 2 provide data on ICH for phase 2 and phase 3 trials, respectively. Of 9 phase 2 trials, 5 suggested a reduced risk and 2 suggested an increased risk of ICH with bolus vs infusion thrombolytic therapy (Figure 1), while of 6 phase 3 trials, 3 suggested an increased risk and none suggested a reduced risk of ICH with bolus therapy (Figure 2).
Figure 1. Intracranial Hemorrhage in Phase 2 Randomized Trials of Bolus vs Infusion Thrombolytic Therapy for AMI
For expansion of trial names, see corresponding references. For heterogeneity, P = .21. Numbers for Kawai et al, InTime, and TIMI-10B correspond to the numbers of patients included in the reported analyses, not the numbers of patients randomized. AMI indicates acute myocardial infarction; n/N, sample size/population size; OR, odds ratio; and CI, confidence interval.
Figure 2. Intracranial Hemorrhage in Phase 3 Randomized Trials of Bolus vs Infusion Thrombolytic Therapy for AMI
For expansion of trial names, see corresponding references. For heterogeneity, P = .12. AMI indicates acute myocardial infarction; n/N, sample size/population size; OR, odds ratio; and CI, confidence interval.
The pooled OR for ICH from meta-analysis of the phase 2 trials (OR, 0.53; 95% CI, 0.27-1.01; P = .06) was both quantitatively and directionally different from the pooled OR obtained from meta-analysis of the phase 3 trials (OR, 1.25; 95% CI, 1.06-1.49; P = .01). There was statistically significant heterogeneity between the phase 2 and phase 3 trials for ICH (P = .01) (Figure 3).
Figure 3. Intracranial Hemorrhage, Stroke, Death, and Reinfarction in Phase 2 and Phase 3 Randomized Trials of Bolus vs Infusion Thrombolytic Therapy for AMI
Heterogeneity for ICH outcome: P = .01 (phase 2 vs phase 3 trials); P = .03 (all trials). For stroke, death, and reinfarction outcomes, no significant heterogeneity between phase 2 and phase 3 trials. Stroke outcome includes nonhemorrhagic stroke and stroke of undetermined cause. AMI indicates acute myocardial infarction; n/N, sample size/population size; OR, odds ratio; CI, confidence interval; and ICH, intracranial hemorrhage.
Meta-analysis of all the trials revealed a 1.0% incidence of ICH with bolus thrombolytic therapy (374/38 932) compared with 0.8% (222/27 697) for infusion thrombolytic therapy (absolute increase of 2 events for every 1000 patients treated: OR, 1.19; 95% CI, 1.01-1.41; P = .04). There was, however, strong statistical evidence of heterogeneity for ICH among the trials (χ2statistic, 26.0; P = .03). Based on the random-effects model, the increase in risk of ICH with bolus thrombolytic therapy was no longer statistically significant (OR, 1.15; 95% CI, 0.88-1.52).
A funnel plot of effect size (OR for ICH) vs study precision (Figure 4) demonstrated a left skew, indicating a disproportionately greater number of trials with point estimates for ICH lower than the overall summary point estimate (10 trials) and a relative paucity of trials demonstrating an increased risk of ICH with bolus thrombolytic therapy (5 trials). However, formal statistical tests for publication bias, including the regression asymmetry test of Egger et al4 (intercept, 0.72; 95% CI, − 0.36 to 1.80; P = .21) and the rank-correlation test of Begg and Mazumdar13 (z = − 1.24; P = .22), were not significant.
Figure 4. Funnel Plot of Effect Size (ICH) vs Precision in Randomized Trials of Bolus vs Infusion Thrombolytic Therapy for AMI
Dashed line indicates overall summary point estimate = 1.19. ICH indicates intracranial hemorrhage; AMI, acute myocardial infarction.
Efficacy Outcomes: Other Stroke, Death, and Reinfarction
Data on stroke due to other causes, death, and reinfarction are presented in Figure 3. Summary ORs from the phase 2 studies for each of these outcomes were less than 1, while for the phase 3 trials they approximated 1. However, there was complete overlap of the 95% CIs of the point estimates for both phase 2 and phase 3 trials for each of these outcomes, with no statistical evidence of heterogeneity between phase 2 and phase 3 trials.
Subgroup Analyses
Subgroup analyses are presented in Figure 5. There appeared to be a lower OR with bolus for ICH in (1) trials that restricted inclusion to patients aged 75 years or younger vs trials including patients older than age 75 years (heterogeneity, P = .07); (2) trials with baseline mean systolic blood pressure below vs above the mean value for all the trials (P = .03); (3) trials with baseline mean patient weights below vs equal to vs above the mean value for all the trials (P = .10); (4) trials using fibrin-specific vs non–fibrin-specific agents (P = .13); (5) open-label vs double-blind trials (P = .16); and (6) trials using an on-treatment vs an intention-to-treat analysis (P = .09).
Figure 5. Odds Ratio for ICH in Subgroups of Trials
Heterogeneity across subgroups: age, P = .07; systolic blood pressure, P = .03; weight, P = .10; fibrin specificity, P = .13; open-label vs double-blind, P = .16; and efficacy vs intention-to-treat, P = .09. ICH indicates intracranial hemorrhage; SBP, systolic blood pressure.
COMMENT
We found wide disparity in estimates of the risk of ICH with bolus vs infusion thrombolytic therapy from meta-analysis of phase 2 trials vs phase 3 trials conducted during the 1990s. The phase 2 trials indicated a statistically nonsignificant reduction in the risk of ICH with bolus thrombolytic therapy, while meta-analysis of the phase 3 trials revealed a statistically significant 25% increase in risk. By contrast, there was no disagreement between phase 2 and phase 3 trials for efficacy outcomes including other causes of stroke, death, and reinfarction.
Previous studies examining disagreement between meta-analysis of small trials and large trials have reported an approximate 80% directional agreement in the pooled point estimates obtained from the small and the large studies.5-6,9 However, these studies focused primarily on efficacy outcomes, and did not consider disagreement for uncommon but serious safety outcomes. Our study has identified uncommon but clinically important safety outcomes as a hitherto unrecognized, and potentially important, cause of disagreement between small (phase 2) and large (phase 3) trials.
We considered a number of possible explanations for disagreement between phase 2 and phase 3 trials for the outcome of ICH. First, a difference in patient characteristics between phase 2 and phase 3 trials may have contributed to disagreement for this safety outcome. Phase 2 trials enrolled a narrower spectrum of patients who may have been at lower risk of ICH with bolus thrombolytic therapy than patients enrolled in phase 3 trials. The additional spectrum of patients included in phase 3 trials, particularly older patients with higher baseline blood pressures and lower body weights, may have increased the likelihood of detecting a true excess in risk of ICH with bolus thrombolytic therapy in these trials.
Second, differences in the intensity of the intervention, with use of lower doses of bolus thrombolytic agent in the phase 2 trials, may have reduced the risk of ICH with bolus agents in phase 2 trials compared with phase 3 trials. Various doses of lanoteplase,26 reteplase,17 and saruplase23 were evaluated in phase 2 trials but, in each case, only the highest dose was used in subsequent phase 3 trials performed with these agents.16, 20, 24, 30 However, the small numbers of patients treated in phase 2 trials and the lack of relevant data from patients treated with lower doses of bolus agent in these studies makes it unclear whether lower doses were associated with a lower risk of ICH.
Third, a greater proportion of phase 2 trials were open-label, and reporting bias in these studies may have contributed to disagreement between phase 2 and phase 3 trials because unblinded investigators are more likely to be biased observers. However, ICH is almost invariably an irreversible and clinically devastating complication that is unlikely to be sensitive to reporting bias.
Fourth, publication bias, as suggested by funnel plot asymmetry for ICH, may have contributed to disagreement between phase 2 and phase 3 trials. Publication bias implies that investigators (or sponsoring pharmaceutical companies) may have preferentially submitted studies with more favorable results (lower risk of ICH) for publication, and that referees may have recomended, and journal editors may have preferentially accepted and published, such studies. English-language bias or citation bias may also have contributed to funnel plot asymmetry since studies with less favorable outcomes are less likely to be reported in English-language journals or cited in the literature,43 and are, therefore, also less likely to be identified in a search of the literature. Although we did not find evidence of publication bias in formal statistical testing, these tests have limited statistical power and cannot exclude publication bias.
The ability of our study to clearly identify a cause of disagreement between phase 2 and phase 3 trials for safety outcomes is limited by a lack of relevant information from all the trials, the likelihood of confounding factors (eg, older age is associated with higher systolic blood pressure), and the uncertain role of publication bias which may be difficult to separate from other causes of heterogeneity that can alter the observed relationship between effect size and study precision. However, our study identified differences between phase 2 and phase 3 trials that appear also to be either important predictors of ICH (older age, higher blood pressure, lower body weight) or possible causes of disagreement between phase 2 and phase 3 trials (open-label design, the likelihood of publication bias). Therefore, these differences may have contributed to the apparently selective disagreement between phase 2 and phase 3 trials for safety but not efficacy outcomes.
The possibility that the apparent differences in ICH risk between phase 2 and phase 3 trials are simply due to chance cannot be excluded. However, even if this were true, chance is much less likely to affect the results of large trials (560 ICHs) than small trials (36 ICHs). Therefore, irrespective of the precise reason, apparently promising results for major, but uncommon, safety outcomes observed in a meta-analysis of small phase 2 trials can be misleading.
We pooled results from trials using different thrombolytic agents and did not consider separately trials performed with the same agent. However, although there may be differences among individual thrombolytic agents with respect to their pharmacological properties (eg, their degree of fibrin specificity), the similar results of large phase 3 clinical trials performed with different bolus agents suggest that these differences are of little clinical relevance. Combining the results of studies using different thrombolytic agents has previously led to meaningful and important advances in our understanding of the role of thrombolytic therapy in AMI,44-45 and we believe that our approach in combining results from trials that used different thrombolytic agents is valid.
Our findings may have implications for the interpretation of meta-analyses based on phase 2 trials in situations where there is potential for uncommon but serious adverse safety outcomes. Even when methodologically rigorous, phase 2 trials remain more susceptible to publication bias and chance than phase 3 trials, while their results may not be generalizable due to the narrower spectrum of patients that are enrolled. Our study suggests that the potential for disagreement is further enhanced when there is potential for serious adverse safety outcomes, particularly since phase 2 randomized trials are unlikely to be powered to detect differences for uncommon safety outcomes such as ICH. Further empirical evidence from studies examining disagreement between phase 2 and phase 3 trials for safety outcomes is required to confirm our findings and to explore possible underlying mechanisms for such disagreement.
Acknowledgments
Author Contributions: Drs Eikelboom and Mehta participated in study concept and design, acquisition of data, analysis and interpretation of data, drafting of the manuscript, critical revision of the manuscript for important intellectual content.
Ms Pogue participated in study concept and design, critical revision of the manuscript for important intellectual content, and provided statistical expertise.
Dr Yusuf participated in study concept and design, analysis and interpretation of data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, provided administrative, technical, or material support, and supervised conduct of the study.
Funding/Support: Drs Eikelboom and Mehta are recipients of Research Fellowship Awards from the Heart and Stroke Foundation of Canada. Dr Yusuf is the recipient of the Medical Research Council of Canada Senior Scientist Award and holds a Heart and Stroke Foundation of Ontario Research Chair.
Acknowledgment: We are grateful to Madhu Mazumdar, PhD, for assistance with statistical testing for funnel plot asymmetry.
Corresponding Author and Reprints: John W. Eikelboom, MBBS, McMaster University, Hamilton General Hospital–McMaster Clinic, 237 Barton St E, Hamilton, Ontario, Canada L8L 2X2 (e-mail: johne{at}ccc.mcmaster.ca).
