The Internal Validity of External Validity: Using Experiments to Validate Three Approaches to Extrapolating Causal Inferences Beyond the Cutoff in Regression Discontinuity

Mathematica Policy Research Inc, Princeton NJ

Investigators

Abstract

The Promoting Research and Innovation in Methodologies for Evaluation (PRIME) program seeks to support research on evaluation with special emphasis on: (1) exploring innovative approaches for determining the impacts and usefulness of STEM education projects and programs; (2) building on and expanding the theoretical foundations for evaluating STEM education and workforce development initiatives, including translating and adapting approaches from other fields; and (3) growing the capacity and infrastructure of the evaluation field. Three types of proposals will be supported by the program: Exploratory Projects that include proof-of-concept and feasibility studies; more extensive Full-Scale Projects; and workshops and conferences. The proposed research attends carefully to item 1 above. In addition, the dissemination plan also attends to item 3. Randomized control trials in STEM education are considered the gold standard for causal inference. Often these models are overly restrictive in practice (for ethical and other reasons). Moreover, many such studies collapse as the research progresses. For example, senior school leadership changes and a school is withdrawn from a study. When this occurs the randomizing process is compromised and so are the resulting inferences. The proposed research will assess the validity and robustness of new approaches for using the regression discontinuity design (RDD) to evaluate educational interventions affecting STEM outcomes. This study will systematically compare and contrast the new approaches for generalizing RDD, assessing how well they produce unbiased causal estimates away from the cutoff score. The research will be conducted using 12 heterogeneous datasets from past educational interventions that (a) were originally evaluated through an RE and (b) included math achievement as an outcome measure. Using the within-study comparison method (Cook et al. 2008), this study will assess the extent to which the RE and a given RDD approach produce similar causal estimates away from the cutoff for the same population. Analyzing multiple datasets with STEM outcomes will provide information about the robustness of the new RDD methods across different types of interventions and student populations.