Evaluation for Actionable Change: A Data-Driven Approach

North Carolina State University, Raleigh NC

Investigators

Abstract

This proposal was submitted in response to the Promoting Research and Innovation in Methodologies for Evaluation (PRIME) solicitation NSF 15-540. The PRIME program seeks to support research on evaluation with special emphasis on exploring innovative approaches, building on and expanding the theoretical foundations, and growing the capacity and infrastructure of the evaluation field. Increasing pressures for accountability have resulted in a push for rigorous evaluation of educational programs and practice. Yet rigorous evaluations such as Randomized Control Trials (RCTs) are expensive and often show small effects. Even RCTs of widely-adopted digital learning platforms can show disappointing results and these results have little impact on subsequent adoptions of programs already entrenched in the educational landscape. New methods are needed to both estimate effects and to indicate ways of improving outcomes for already-adopted digital learning tools. With platforms currently in wide-scale use, novel approaches to assessing use patterns and their relations with outcomes can both evaluate maximal effectiveness and provide means for improved effectiveness. This research will develop an evaluation of ST Math, a K-8 digital learning platform designed to strengthen the mathematical competency of students through enhancing both their understanding of math concepts and their motivation for math learning. By investing in evaluation innovations to improve digital learning platforms such as ST Math, such programs could reach large numbers of children with maximal effectiveness with each iteration. Development of automated tools for improvement can also enhance both the efficiency and efficacy of the digital learning platforms. This work will be accomplished through a partnership between researchers at North Carolina State University (NC State) with expertise in educational evaluation, educational data mining, and assessment with program developers at the non-profit MIND Research Institute (MIND). This project will explore novel, and noninvasive, approaches for determining the impact of the ST Math digital learning environment. It will advance the analytical basis for formative assessment using process data and build algorithms that improve STEM teaching and learning by facilitating the automatic recognition of teachable moments for learning. The transformative potential of this research resides in the creation of new cross-disciplinary approaches that can be used to not only evaluate impact, but to inform improved teaching and learning in STEM, by leveraging observed behaviors of students and teachers and educational data mining techniques. Specifically, this research will explore novel methods for detecting, visualizing, and evaluating students? puzzle-solving and puzzle-selection behaviors. The PIs will assess whether the detected patterns are driven by students? incoming competence or can be used to predict their short and long-term performance. By linking student and teacher behavior patterns with important learning and motivational outcomes, the researchers may be able to recommend promising actions to teachers and potential refinements to developers. This work has the potential to not only transform the use and success of the ST Math platform, but to create methods that can be refined and transferred to the evaluation and implementation of other platforms.