Creating a Theoretical and Empirical Foundation for Better Non-Equivalent Control Group Designs in STEM Research

George Washington University, Washington DC

Investigators

Abstract

This proposal was submitted in response to EHR Core Research (ECR) program announcement NSF 15-509. The ECR program of fundamental research in STEM education provides funding in critical research areas that are essential, broad and enduring. EHR seeks proposals that will help synthesize, build and/or expand research foundations in the following focal areas: STEM learning, STEM learning environments, STEM workforce development, and broadening participation in STEM. The ECR program is distinguished by its emphasis on the accumulation of robust evidence to inform efforts to (a) understand, (b) build theory to explain, and (c) suggest interventions (and innovations) to address persistent challenges in STEM interest, education, learning, and participation. Random assignment of a STEM innovation to participants (students, teachers, schools) affords the researcher the opportunity to estimate an unbiased estimate of the effect of an intervention. However, random assignment may not be possible for ethical and practical reasons. Consequently, it is often difficult to build statistical knowledge of the impact of a STEM innovation without introducing bias into the process. The research to be conducted here will build and test a framework that will allow researchers to minimize statistical bias and better understand the impacts of STEM innovations (e.g., a new curriculum, a new way of teaching, or a change in policy) in settings where random assignment is not possible. This proposal uses advances in statistical theory and past empirical findings to craft testable hypotheses that, if validated, will improve causal hypothesis testing in STEM education research. The main hypotheses speak to reducing, and perhaps eliminating, the selection bias in quasi-experiments. The proposal focuses on the bias-reducing role of three design elements: a comparison group that is local to the treatment group, a pretest measure of the study outcome, and a rich set of pre-intervention covariates that are multi-dimensional (cover more than one substantive domain), multi-temporal (cover more than one time point) and multi-level (available at both the student and school levels). Each of these elements often reduces some bias and sometimes eliminates it entirely. How do they combine to reduce bias? The proposal will test (1) which combinations of the elements reduces most bias; (2) how often the bias reduces to close to zero; (3) do they do so robustly across STEM datasets with diverse interventions and students; and (4) is bias inadvertently introduced, rather than reduced, when the 3 design elements are combined. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.