Findings from Empirical Within Study Comparisons about the Role of Pretests and Proxy Pretests in Adjusting for Selection Bias in STEM Quasi-Experiments

Northwestern University, Evanston IL

Investigators

Abstract

Experimental designs in education that include treatment and control groups that are randomly assigned to different conditions of an intervention are considered the most rigorous choice to be able to evaluate causal claims. However, a randomized controlled trial (RCT) is frequently not feasible in developing evidence around the impact of many STEM practices, programs and policies. Researchers from Northwestern University examine how quasi-experimental designs that include treatment and comparison groups and outcome measures, but that do not include the random assignment to treatment and control conditions, might produce the same quality of findings as experimental designs. The researchers are also examining if pretreatment measures of study outcomes might be replaced with proxy pretests, measures in the same domain as an original pretest but in a different form, to replicate the findings from experimental studies using the original pretest. Many STEM research studies frequently have access in quasi-experimental studies for data from these proxy pretests in archival datasets, such as the state longitudinal data systems. Researchers in this project have identified a number of RCTs that they use to examine whether changing the nature of the comparison group, from the original randomly assigned group, to one that is statistically adjusted from a larger population will show similar causal estimates as the original RCT. Using a within-study comparison, the researchers determine the differences in the RCT and quasi-experimental findings. They are studying the use of proxy pretreatment tests, such as math achievement data from state longitudinal data systems, to examine the extent to which these measures replicate findings in the original RCTs. They also examine the effect that modeling with a number of pre-intervention covariates has on causal estimates. If the differences between the causal estimates in the RCT and the new quasi-experimental study are low, then the quasi-experimental design is adequate to measure impact. Large data sets are increasingly available in education, especially with the development of district and local data systems that expand the information gathered about students, teachers and schools. These data sets provide opportunities for research and evaluation studies that can operate with population level data rather than samples of data drawn randomly from the population. The determination of the quality of quasi-experimental evaluation designs that result from this study provides information that can be used by policy makers and researchers to study questions about educational treatments, programs and policies that have been intractable to study. Removing the potential barrier of random assignment, while still maintaining the quality of an RCT, expands the methodological toolkit of evaluators.