Collaborative Research: Developing Methodological Foundations for Empirical Evaluations of Non-Experimental Methods in STEM Intervention Evaluations

University Of Virginia Main Campus, Charlottesville VA

Investigators

Abstract

The Promoting Research and Innovation in Methodologies for Evaluation (PRIME) program seeks to support research on evaluation with special emphasis on: (1) exploring innovative approaches for determining the impacts and usefulness of STEM education projects and programs; (2) building on and expanding the theoretical foundations for evaluating STEM education and workforce development initiatives, including translating and adapting approaches from other fields; and (3) growing the capacity and infrastructure of the evaluation field. Three types of proposals will be supported by the program: Exploratory Projects that include proof-of-concept and feasibility studies; more extensive Full-Scale Projects; and workshops and conferences. The proposed research attends carefully to item 1 above. This research will establish a coherent framework for the design, implementation, and analysis of within-study comparisons for evaluating non-experimental methods. It will also establish an infrastructure for conducting an ongoing quantitative synthesis of results from within-study comparison designs. This research study will develop the methodological foundations for using within-study comparison designs (WSCs) to evaluate non-experimental methods in STEM evaluation settings. In WSC designs, treatment effects from a non-experiment are compared to those produced by a randomized experiment that shares the same target population. The purpose of a WSC is to determine whether the non-experiment can replicate results from an experimental benchmark, and the contexts and conditions under which these methods perform well in field settings. The project has two overarching research aims.