Assessing the Impact of Type I and Type II Errors on High Strength of Evidence Outcomes

Research Triangle Institute, Durham NC

Investigators

Linked publications & trials

Abstract

PROJECT SUMMARY/ABSTRACT DESCRIPTION: See instructions. This must contain a summary of the proposed activity suitable for dissemination to the public (no proprietary/confidential information). It should be a self-contained description of the project and contain a statement of objectives and methods to be employed. It should be informative to other persons working in the same or related fields. DO NOT EXCEED THE SPACE PROVIDED. Background: Over the past decade, systems to grade the strength of evidence (SOE) have become commonly used tools to convey certainties and uncertainties in bodies of evidence synthesized in systematic reviews. For decision makers, SOE grades are crucial because they predict the impact of future research on observed estimates of effect. In a recent methods project, we determined that the predictive value of SOE grades is low. Specifically, more than 20% of outcomes graded high SOE substantially changed in magnitude of effect as new studies were published. Because ?high SOE,? by definition, conveys strong confidence in the correctness of estimates, such a high proportion of changing estimates raises concerns for clinical and policy decision making. To date, the reasons for the low predictive value of SOE grades have not been determined. Specific Aims: The goal of our proposed research project is to identify factors that are responsible for the limited predictive value of SOE grades. The specific aims of our proposed research are: 1. To determine the proportion of high SOE outcomes with an increased risk of type I or type II errors. 2. To determine whether increased risks of type I or type II errors in high SOE outcomes can be attributed to inappropriate grading (i.e., a given outcome should not have been graded high SOE following current guidance) or to flaws in the conceptual approach that did not detect increased risks for statistical errors. Methods: We will sample 100 high-SOE outcomes from Cochrane and Agency for Healthcare Research and Quality (AHRQ) reports. We will focus on dichotomous outcomes (graded high SOE) that were based on meta-analyses of randomized controlled trails and for which data are available to reproduce the meta- analyses. To determine the risk for type I and type II errors, we will apply Trial Sequential Analyses (TSA) to determine the risks of type I and type II errors in these bodies of evidence. In cases where high SOE is not supported by TSA results, we will determine reasons for the lack of concordance. Implications: Our results will provide further insight as to whether the limited predictive value of SOE grades can be improved by better guidance and training or whether the conceptual framework for SOE needs to be substantially rethought and revised and, if so, in what ways.