NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness
Methods Research Reports
Investigators: Susan L Norris, MD, MPH, MSc, Haley K Holmer, MPH, Lauren A Ogden, BA, Rongwei Fu, PhD, Ahmed M Abou-Setta, MD, PhD, Meera S Viswanathan, PhD, and Melissa L McPheeters, PhD, MPH.
Author Information and AffiliationsStructured Abstract
Objectives:
The objectives of this exploratory study were to: (1) describe the frequency of selective outcome reporting (SOR) and selective analysis reporting (SAR) in randomized controlled trials (RCTs) included in reviews of comparative effectiveness for outcomes of benefit; (2) explore potential predictors for SOR and SAR; and (3) assess the reliability and validity of the Outcome Reporting Bias in Trials (ORBIT) classification system for missing or incomplete outcome reporting.
Data Sources and Methods:
We selected three comparative effectiveness reviews (CERs) funded by the Agency for Healthcare Research and Quality that included drug–drug comparisons. Within each CER, we then specified one outcome that fulfilled explicit criteria (the “index outcome”) and examined the RCTs in the CER that reported that outcome. We then searched trial registries for study registration information and results for each RCT. Using available registry information to complement information in the methods section of the publication, we determined the frequency of SOR and SAR, and we examined prespecified predictors of SOR and SAR. Lastly, using the ORBIT classification of SOR, we attempted to examine the inter-rater reliability of ORBIT and its validity, comparing information contained within the publication to assessments of SOR, using the additional information obtained from trial registries.
Results:
RCTs published in 2005 or later and reporting the index outcome were not consistently listed in trial registries, with 29 percent, 67 percent, and 75 percent of trials registered for each of the three CERs. In addition, publications did not consistently report trial registration. Results were infrequently listed in ClinicalTrials.gov, even after 2008, when reporting became mandatory for certain types of trials. Trial registration frequently occurred after the study was completed (in 25 percent, 50 percent, and 42 percent of trials in each of the three CERs). Changes occurred in the specification of the index outcome in the registry in 42 percent and 17 percent of trials in two CERs (the index outcome in the third CER was never mentioned in the registry). We did not find the ORBIT classification tool particularly useful: it was difficult to implement, and the nine classes were difficult to reliably distinguish. In addition, ORBIT classes did not describe a type of SOR and SAR that we frequently encountered: the addition of outcomes measures, subgroups, and other analyses to published results that were not prespecified in the publication’s methods section or listed in the registry. Finally, trial registries were of little use in identifying SOR unless trial results were listed in the registry and of no use in identifying SAR.
Conclusions:
We identified numerous challenges in identifying and characterizing SOR and SAR in this pilot study of three CERs. Existing tools were suboptimal: ORBIT does not encompass the type of SOR and SAR where results in the publication were not prespecified in the methods section or in the registry. The design of our study (focusing on RCTs with results in the CER) precluded identifying certain types of SOR where the outcomes were not reported at all in the study. The presentation and content of ClinicalTrials.gov could be improved to better assist the systematic reviewer in identifying potential SOR and SAR. Further research is needed to develop efficient, tailored approaches to identifying and characterizing SOR and SAR in trials.
Contents
- Preface
- Acknowledgments
- Peer Reviewers
- Background
- Methods
- Overview of Methods
- Selection of the Cohort of Comparative Effectiveness Reviews
- Identification of an Index Outcome for Each Included Comparative Effectiveness Review
- Identification of Randomized Trials Reporting the Index Outcome Within Each Comparative Effectiveness Review
- Selection of Specific Comparative Effectiveness Reviews for This Pilot Study
- Identification of Trial Registration for Randomized Controlled Trials Reporting the Index Outcome
- Data Abstraction
- Predictors of Selective Outcome Reporting and Selective Analysis Reporting
- Assessment of the Reliability and Validity of the ORBIT Classification of Selective Outcome Reporting
- Data Syntheses and Analyses
- Results
- Discussion
- Strengths and Limitations of This Study
- Challenges Encountered Developing Our Study Protocol
- Challenges and Recommendations Regarding Methods and Available Tools for Exploring Selective Outcome Reporting and Selective Analysis Reporting
- Suggestions for Identifying and Characterizing Selective Outcome Reporting and Selective Analysis Reporting
- Conclusions
- References
- Abbreviations and Acronyms
- Appendixes
Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No. 290-2007-10057-I, Prepared by: Oregon Evidence-based Practice Center, Portland, OR
Suggested citation:
Norris SL, Holmer HK, Ogden LA, Fu R, Abou-Setta AM, Viswanathan MS, McPheeters ML. Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-10057-I.) AHRQ Publication No. 12-EHC110-EF. Rockville, MD: Agency for Healthcare Research and Quality. August 2012. www.effectivehealthcare.ahrq.gov/reports/final.cfm.
This report is based on research conducted by the Oregon Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10057-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.
This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.
None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.
- 1
540 Gaither Road, Rockville, MD 20850; www
.ahrq.gov
- NLM CatalogRelated NLM Catalog Entries
- Clinical trial registries are of minimal use for identifying selective outcome and analysis reporting.[Res Synth Methods. 2014]Clinical trial registries are of minimal use for identifying selective outcome and analysis reporting.Norris SL, Holmer HK, Fu R, Ogden LA, Viswanathan MS, Abou-Setta AM. Res Synth Methods. 2014 Sep; 5(3):273-84. Epub 2014 Mar 11.
- Issues relating to selective reporting when including non-randomized studies in systematic reviews on the effects of healthcare interventions.[Res Synth Methods. 2013]Issues relating to selective reporting when including non-randomized studies in systematic reviews on the effects of healthcare interventions.Norris SL, Moher D, Reeves BC, Shea B, Loke Y, Garner S, Anderson L, Tugwell P, Wells G. Res Synth Methods. 2013 Mar; 4(1):36-47. Epub 2012 Nov 13.
- Review Making protocols available with the article improved evaluation of selective outcome reporting.[J Clin Epidemiol. 2018]Review Making protocols available with the article improved evaluation of selective outcome reporting.Calméjane L, Dechartres A, Tran VT, Ravaud P. J Clin Epidemiol. 2018 Dec; 104:95-102. Epub 2018 Sep 6.
- Review Screening for Chronic Obstructive Pulmonary Disease: A Systematic Evidence Review for the U.S. Preventive Services Task Force[ 2016]Review Screening for Chronic Obstructive Pulmonary Disease: A Systematic Evidence Review for the U.S. Preventive Services Task ForceGuirguis-Blake JM, Senger CA, Webber EM, Mularski R, Whitlock EP. 2016 Apr
- Review A Process for Robust and Transparent Rating of Study Quality: Phase 1[ 2011]Review A Process for Robust and Transparent Rating of Study Quality: Phase 1Ip S, Kitsios GD, Chung M, Lau J. 2011 Nov
- Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effect...Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness
Your browsing activity is empty.
Activity recording is turned off.
See more...