 |
|
Association Between Unreported Outcomes and Effect Size Estimates in Cochrane Meta-analyses
To the Editor: The selective reporting of outcomes in randomized controlled trials (RCTs), or outcome reporting bias, has been identified,1 but its prevalence and potential effect in meta-analyses is unknown. We investigated (1) to what extent Cochrane meta-analyses include only a proportion of identified RCTs when estimating treatment effects on patient-important outcomes and (2) whether the proportion of RCTs included in a meta-analysis is associated with its pooled effect size.
Methods
| |
Issue 4 of the Cochrane Library, 2005, contains 2524 reviews. Analysis of the first 56 meta-analyses found 10 that met the eligibility criterion of containing 10 or more RCTs. A power calculation based on their regression coefficients indicated that detecting statistically significant coefficients of this magnitude required approximately 100 reviews. We therefore selected 500 reviews by random-number generator for assessment.
Two investigators independently identified reviews with 10 or more RCTs. For each review, 3 investigators selected 2 dichotomous and 2 continuous outcomes of greatest patient importance based on the review authors' descriptions; if primary outcomes were not specified, this was based on our assessment. Disagreement was resolved through discussion. Investigators extracted the number of identified RCTs, the number of RCTs that contributed data to the meta-analysis, and the pooled random-effects model odds ratio (OR) or standardized mean difference (SMD). The proportion of studies that contributed to each meta-analysis was calculated, and a linear regression model to examine how these proportions were associated with ORs and SMDs was constructed. Nonlinear effects were explored with a quadratic model, but the quadratic term was not statistically significant and goodness of fit was poorer, so these results are not reported. Means and 95% confidence intervals (CIs) were calculated for the ORs and SMDs for each stratum, representing increments of 20% in the proportions of contributing RCTs. Statistical analysis was performed using SPSS version 11.5 (SPSS Inc, Chicago, Ill).
Results
The eligibility criterion was met by 156 reviews that included 4222 RCTs (range, 10-217 per review; interrater agreement  = 0.96). A total of 227 dichotomous and 167 continuous outcomes were chosen. (A list of the selected reviews and outcomes is available from the authors.) A median of 46% (interquartile range, 20%-75%) of the identified RCTs in each meta-analysis contributed to the pooled estimates. When outcomes favored the intervention, the magnitude of effect generally decreased as the proportion of contributing trials increased (Table). The regression coefficients were statistically significant; for OR measures,  = –0.16 (95% CI, –0.29 to –0.01; P = .03) and for SMD measures, = –0.18 (95% CI, –0.35 to –0.01; P = .04). For each 10-point decrease in the percentage of contributing trials, the OR increased by 1.046 (95% CI, 1.004-1.090) and the SMD increased by 0.041 (95% CI, 0.002-0.079). When outcomes favored the control condition, regression coefficients were 0.16 (95% CI, –0.16 to 0.45; P = .33) and –0.08 (95% CI, –0.37 to 0.22; P = .59), respectively.
|
|
|
|
Table. Relationships Between Percentage of Contributing RCTs and Effect Size Estimates When Meta-analyses Favored Intervention
|
|
|
Comment
On average, approximately half of the RCTs identified by Cochrane reviews did not contribute to the pooled effect size estimates for patient-important outcomes in their meta-analyses. There appears to be an inverse association between percentage of RCTs contributing and effect size.
There are 3 possible explanations for incomplete inclusion of outcomes. First, studies in which the outcome is unreported may not have measured that outcome. Second, these studies may have measured the outcome, but the investigators did not report the results for reasons unrelated to the findings, such as preferences of the funder or journal. Third, investigators or journal editors may have selectively failed to report some of the measured outcomes because their results were not of great magnitude. The last explanation is supported by our findings.
Limitations of this study must be considered. We did not measure independent agreement in the selection of the outcomes of focus. The associations were modest and of borderline statistical significance. We did not assess potential confounders or explanatory variables, such as funding source, journal, types of outcomes, stage in development cycle, or statistical power. Further research exploring these factors may provide insights into these findings.
Pending this, investigators and clinicians should be aware that the results of meta-analyses may be overestimating the true underlying effects when the proportion of contributing trials is low. Potential reporting bias would be reduced by trialists preregistering their study protocols and reporting all outcomes measured2 and by agreeing on a standard set of outcomes for interventions in a given field.3
Author Contributions: Dr Furukawa had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Furukawa, Watanabe, Montori, Guyatt.
Acquisition of data: Furukawa, Watanabe, Omori.
Analysis and interpretation of data: Furukawa, Watanabe, Omori.
Drafting of the manuscript: Furukawa, Watanabe, Omori.
Critical revision of the manuscript for important intellectual content: Furukawa, Watanabe, Omori, Montori, Guyatt.
Financial Disclosures: Dr Furukawa reports that he has received research funds and speaking fees from Asahi Kasei, Astellas, Dai-Nippon, Eisai, Eli Lilly, GlaxoSmithKline, Janssen, Kyowa Hakko, Meiji, Organon, Pfizer, Tsumura, Yoshitomi, and Zelia and that the Japanese Ministry of Education, Science, and Technology and the Japanese Ministry of Health Labor and Welfare have also funded his research. Dr Watanabe reports that he received a grant from the Cochrane Child Health Field bursary scheme. Dr Montori reports receiving research funding from the Mayo Foundation, the Endocrine Society, and the American Diabetes Association. Dr Guyatt reports that he has received grant funding from Pfizer, Lotte and John Hecht Foundation, Bristol-Myers Squibb, Aventis, Pharmacia, Leo Pharma, AstraZeneca AB, and Eli Lilly Canada. He has also received consultation fees from Up-to-Date. All authors have written Cochrane protocols and/or reviews. Dr Furukawa is an editor of 1 Cochrane review group and Dr Guyatt is a co-convener of 2 Cochrane methods groups. No other disclosures were reported.
Funding/Support: No external funding was used for this study.
Toshi A. Furukawa, MD, PhD
furukawa{at}med.nagoya-cu.ac.jp
Norio Watanabe, MD;
Ichiro M. Omori, MD
Department of Psychiatry and Cognitive-Behavioral Medicine
Nagoya City University Graduate School of Medical Sciences
Nagoya, Japan
Victor M. Montori, MD, MSc
Knowledge and Encounter Research Unit
Department of Medicine
Mayo Clinic College of Medicine
Rochester, Minn
Gordon H. Guyatt, MD, MSc
Department of Clinical Epidemiology and Biostatistics
McMaster University
Hamilton, Ontario
1. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291:2457-2465.
FREE FULL TEXT
2. DeAngelis CD, Drazen JM, Frizelle FA, et al, Committee on Outcome Measures in Rheumatoid Arthritis Clinical Trials. Is this clinical trial fully registered? a statement from the International Committee of Medical Journal Editors. JAMA. 2005;293:2927-2929.
FREE FULL TEXT
3. Felson DT, Anderson JJ, Boers M, et al. The American College of Rheumatology preliminary core set of disease activity measures for rheumatoid arthritis clinical trials. Arthritis Rheum. 1993;36:729-740.
ISI
| PUBMED
Letters Section Editor: Robert M. Golub, MD, Senior Editor.
JAMA. 2007;297:468-470.
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
Review: TCAs, anticonvulsants, opioids, and capsaicin cream are effective for diabetic neuropathy
Murad and Smith
Evid. Based Med. 2008;13:21-21.
FULL TEXT
|
