Rigorous Measures of Implementation: A Suite of Tools for Evaluating STEM Instructional Materials Use

University Of Chicago, Chicago IL

Investigators

Abstract

The Center for Elementary Mathematics and Science Education (CEMSE) at the University of Chicago will conduct a full-scale 2-year study to demonstrate and further develop and test the theory upon which the suite of Fidelity of Implementation (FOI) instruments is based by exploring the practical use of these tools. This mixed-method study grows out of CEMSE's past NSF-funded studies focusing on the development of innovative tools for evaluating the implementation of reform-based K-8 mathematics and science instructional materials programs. The goals of this project are: 1) operationalizing the process of instrument adaptation to a locally-developed science program; 2) further refining and developing the FOI instruments through field testing; 3) exploring methodologies for instrument use in evaluating the effect of program implementation on student outcomes; and 4) sharing, disseminating, and communicating knowledge about measurement of implementation of science and mathematics instructional materials through both online and site-based collaborative discussions with practitioners, evaluators and researchers. This project will involve three mid-sized school districts located in Champaign, IL, Naperville, IL, and Newton, MA, that use Everyday Mathematics (EM) and a reform-based science curriculum. Within the participating districts, the investigators will use a purposeful sampling strategy to select 17 to 20 elementary schools that best represent the district in terms of school size, student demographics, and/or student achievement. In year 1, the existing FOI instruments for EM will be administered to all K-5 teachers and school leaders in the participating schools. The instruments will also be adapted, field tested, and revised for measurement of implementation of the science curriculum. In Year 2, a revised set of FOI instruments for science will be administered to participating teachers and school leaders. Data analysis will include exploring the status and types of EM and science materials implementation. Standardized state test scores and grades will be collected within all participating classrooms to link status and types of instructional materials use to student achievement. Underlying this study is a belief that efforts to improve evaluation of the implementation of educational innovations are only truly effective when evaluation tools and data are usable and useful to the audiences for which they are intended. There exists a continuing need in the field for research instruments that describe, measure, and explain specific aspects of mathematics and science instructional materials implementation in order to understand their effectiveness and to inform ongoing improvement efforts in STEM education. The need for more rigorous measurement of mathematics and science instructional materials is also particularly relevant to school and district leaders seeking to bring effective programs to their schools. These FOI instruments will enable stakeholders to obtain data at a level of specificity that can inform ongoing improvements and identify elements of their curricula that are leading to the desired practices and outcomes. This study will ultimately contribute to the broader field of STEM education by furthering an understanding of the ways in which an innovative approach to FOI measurement may be applied in school settings while creating and sharing knowledge about the evaluation of implementation of STEM instructional materials.