Statistical and Psychometric Methods for Measuring the Extent to Which Culturally Responsive Assessments Reduce Cultural Bias

Educational Testing Service, Princeton NJ

Investigators

Abstract

This research project will develop a set of statistical tools to analyze data from culturally responsive assessments. Culturally responsive assessments have emerged as a potential solution to the concern that standardized assessments perpetuate social injustice through their content, design, and use. However, few true culturally responsive assessments exist, and there is a lack of proven statistical methods to analyze data from them. This project will use the tools to be developed and data collected by the project personnel to measure the extent to which culturally responsive assessments reduce cultural bias. Scientific products will be disseminated via social media, workshops, and publications in peer-reviewed journals. Open-source software will be developed to make the tools more accessible to a broader audience. The project will engage graduate students in conducting this research, and the investigators will include students from underrepresented groups in their project. The findings and products originating from this project will be useful for scholars across the social, behavioral, and economic sciences. The project also will have implications for society as it will provide a new set of tools to promote social justice. This research project will provide scholars with new tools for producing fair scores from culturally responsive assessments. The project team first will examine several statistical models and approaches to analyzing data from culturally responsive assessments and demonstrate the effectiveness of these approaches using simulated data. The project team then will develop several culturally responsive assessments with the help of a multidisciplinary team of researchers and practitioners. These assessments will be administered to several thousand examinees obtained from a platform such as Amazon Mechanical Turk or Prolific. The newly developed models and approaches will be used to analyze the data. The analyses will focus on measuring the type and extent of cultural bias in standardized tests and determining the extent to which culturally responsive assessments produce more fair and equitable results. The project will produce publicly available principles and guidelines for preparing culturally responsive assessments and user-friendly software and accompanying manual to implement the approaches developed in this project. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.