CAREER: Complex Causal Moderated Mediation Analysis in Multisite Randomized Trials: Uncovering the Black Box Underlying the Impact of Educational Interventions on Math Performance

University Of Pittsburgh, Pittsburgh PA

Investigators

Abstract

Interventions aimed at improving mathematics performance are frequently implemented in school settings. Multisite randomized trials, or studies in which individuals are randomized to condition within schools, are frequently used to assess the effect of these interventions. The most common question in these studies is whether or not the intervention is effective. While determining whether an intervention is effective is an important question, it does not provide a comprehensive picture of the intervention. Studies should also be designed to determine how an intervention works, or the mechanisms underlying the intervention, known as the mediation mechanisms. In addition, the effect of the mechanism may differ depending on student characteristics, school context, local implementation, or a host of other factors, known as moderated mediation. Assessing moderated mediation is critical in understanding for whom and under what contexts the intervention is effective, and why. Currently, statistical methods and tools do not exist for unpacking complex mediation mechanisms in multisite trials. The purpose of this study is to develop methods and tools to enable researchers to answer questions crucial for unpacking complex mediation mechanisms in multisite randomized trials, including (1) how the total impact of an intervention is mediated by one mediator or multiple concurrent or sequential mediators and (2) how the mechanisms vary by individual and contextual factors. These methods and tools will help researchers better understand the complex effects of interventions so that they can improve and tailor interventions to different individuals and school contexts and thus enhance educational equity. This project will develop analysis procedures for multisite causal moderated mediation analysis with one mediator or multiple concurrent or sequential mediators. The identification of the causal effects relies on the assumption of no unmeasured confounding, which is usually violated in real applications. Therefore, this project will also develop intuitive sensitivity analysis strategies to assess the potential influence of not only unmeasured pretreatment confounding but also posttreatment confounding. The methods will be applied to the National Study of Learning Mindsets (NSLM) and Head Start Impact Study (HSIS) to investigate the mediation mechanisms underlying the impact of educational interventions on math performance and their heterogeneity. Comprehensive Monte Carlo simulations will be conducted to evaluate the performance of the proposed methods. A user-friendly R package with a graphical interface will be developed to enable empirical researchers to generate a new set of thorough, precise, and valid empirical evidence regarding the heterogeneity of causal mediation mechanisms across individuals and contexts. The usability of the package will be tested through simulations, real data analyses, and focus groups. Courses and workshops will be offered to train diverse students and scholars to adopt the analytic framework and tools developed from this project. All the software documentation and training materials will be publicly and freely accessible. This is a Faculty Early Career Development Program project responsive to a National Science Foundation-wide activity that offers the most prestigious awards in support of early-career faculty who have the potential to serve as academic role models in research and education. This project is supported by NSF's EDU Core Research (ECR) program. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field. Investments are made in critical areas that are essential, broad and enduring: STEM learning and STEM learning environments, broadening participation in STEM, and STEM workforce development. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.