SGER: Examining the Feasibility of Developing a "Scalability Index"

Harvard University, Cambridge MA

Investigators

Abstract

This study will examine the feasibility of developing a quantitative index that measures the relative scalability of an innovation. Research findings typically show substantial influence of contextual variables in shaping the desirability, practicality, and effectiveness of interventions. Developing a quantitative strategy for assessing the relative ease with which innovations successful under ideal conditions can scale up into relatively inhospitable settings will aid in many types of educational improvement. Just as estimating standardized effect sizes across published experiments that have used all manner of statistical techniques for ascertaining their original findings has proven a useful way of comparing the relative educational efficacy of different interventions, so a similar cross project index that estimated the relative scalability of innovations is of potential value to individual investigators, to practitioners and to policy makers. Identifying factors within the intervention's context that represent important conditions for success and summarizing the extent to which the effect of the intervention is sensitive to variation in each will provide prospective adopters of the innovation a better sense of what its likely effectiveness would be in their own particular circumstances. An initial step that is essential to creating a viable scalability index is the careful specification of a sensible framework of contextual factors that represent possible general conditions for success of educational innovations. For many types of innovations, a relatively small set of contextual factors are often very influential in determining effectiveness. This project will create such a subset of all possible contextual factors and investigate its utility in real data. At its core, the evaluation of the sensitivity of an intervention's impact to select contextual conditions is a question of statistical interactions. In a single study, such questions are usually addressed by the inclusion in the statistical models of interactions between the treatment and its conditions for success. One approach, then, to the creation of a true scalability index is to ensure that such interactions are included in the statistical models that underpin the data-analyses conducted to assess the implementation of educational interventions. The several effect sizes anticipated for the intervention under each of the interacting conditions can be pooled into a global index of scalability that captures the extent to which the intervention's effect size is sensitive to variation in the conditions for success. However, important technical challenges to implementing this approach in practice may render this method infeasible. The intellectual merit of this proposed activity is its examination of statistical approaches to quantifying scalability. Whether or not a scalability index is feasible, the insights gained by attempting to produce this metric will inform other initiatives to develop generalizable metrics from implementation data. If the scalability index proves valid and is practical given the types of datasets produced by implementations of innovations, its broader impact and significance to the field is threefold: 1. prospective adopters of an innovation will have a better sense of what its likely effectiveness would be in their own particular circumstances, 2. policymakers will have a better sense of which innovations are most promising in their potential for adaptability to a wide range of settings, and 3. researchers can contrast the scalability of various types of innovations, thereby gaining insights into how to improve design and implementation.