CAREER: Design of Multisite Moderation Studies to Examine the Factors that Explain Treatment Effect Variation on Student Mathematics Achievement in Teacher Professional Development

University Of North Carolina At Chapel Hill, Chapel Hill NC

Investigators

Abstract

This is a Faculty Early Career Development (CAREER) project submitted to the EHR Core Research (ECR) program, responsive to Program Solicitation NSF 15-555. Its purpose is to develop a comprehensive framework to inform and investigate treatment effect variation in teacher professional development (PD) studies using multisite moderation studies focused on the variables that affect the direction and magnitude of the relation between the treatment variable and the outcome variable. The study will accomplish four integrated research and education goals. The first goal will develop the statistical theory underlying the effective and efficient design of multisite moderation studies of professional development, and derive formulas for the statistical power and the minimum detectable effect size difference with confidence intervals for moderator effects in two- and three-level multisite randomized and nonrandomized studies. The second goal will apply a framework and statistical theory to empirically estimate the moderator effects and design parameters of existing Elementary Mathematics Specialists (EMS) programs, and demonstrate the calculation of the statistical power and the minimum detectable effect size difference for this EMS program using the formulas derived from the first goal. The third goal will execute the formulas derived from the first doal in the existing free-available software, and develop accompanying software documentation. Finally, the fourth goal will provide graduate students and researchers PD in using the results from this project to design multisite moderation studies. It will include developing curricular and instructional materials on the tools and conducting workshops using these materials. The proposed research builds on data from an ongoing NSF-funded mathematics teacher PD study, which examines the average effects of the EMS certified teachers with the EMS non-certified teachers that are matched within schools or districts. The CAREER study will gather secondary data from the ongoing work on students' mathematics achievement; teachers' mathematical content knowledge; and covariates and potential moderators, such as type of school district, school, teacher, and student background information. Data analyses will focus on students' mathematics achievement gains empirical estimates of moderator effects using two- and three-level hierarchical linear modeling for moderation analyses; empirical estimates of design parameters necessary for power analyses; and demonstration of calculating the power and the minimum detectable effect size difference for this EMS program. Likewise, the project will develop software and documentation that make the complex statistical concepts broadly accessible and practical to applied researchers to implement the work developed from this study; as well as the training materials and design of the professional development workshops. The project will be evaluated both internally and externally through an advisory board and users of the software and PD materials. The main outcome of this effort will be a framework and tools to increase researchers' capacity to identify the factors and conditions that explain the treatment effect variation in teacher professional development; thus, improving teacher effectiveness through the rigorous evidence of research findings in response to questions regarding "work for whom, and under what circumstances", and interpreting and exploring the sources of the treatment effect variations.