Constructing Valid, Equitable, and Flexible Kinematics and Dynamics Assessment Scales with Evidence-Centered Design

West Virginia University Research Corporation, Morgantown WV

Investigators

Abstract

This project aims to serve the national interest by improving teaching and learning in introductory physics courses. This collaborative project involves investigators at Michigan State University (Award DUE-2235518), Ohio State University (Award DUE-2235595), and West Virginia University (Award DUE-2235681). Over the past 25 years, students' learning in introductory physics courses has often been assessed using instruments such as the Force Concept Inventory (FCI) and the Force and Motion Conceptual Evaluation (FMCE), which were designed to measure students' conceptual understanding of Newtonian mechanics. Researchers have since come to understand that these legacy instruments have key features that can be greatly improved, including i) psychometric characteristics that may limit the usefulness of those instruments for both research and instructional applications; ii) demographic biases; and iii) artwork and contexts from a different era reflecting limited diversity. In view of these issues, there is a need to develop new and improved assessment tools focused on the accurate and fair measurement of students' conceptual understanding in introductory physics courses. This project will provide instructors and education researchers the opportunity to flexibly construct research-based assessment instruments. These new instruments will allow instructors not only to determine their students' general understanding of kinematics and dynamics but also to develop a fine-grained picture of that understanding, and this knowledge will allow the instructors to direct resources where they are most needed. To replace the widely used legacy assessments for mechanics (the FCI and the FMCE), the investigators aim to develop a new set of assessments composed of items and subscales that are both research-informative and instructionally informative and do not have psychometric limitations or demographic biases. They will leverage the Evidence-Centered Design (ECD) framework to gather broad input from the physics community and construct an array of valid, fair, and flexible items organized into subscales. The items will be extensively validated and tested with approximately 15,000 students at six universities to ensure superior psychometric properties and instrumental fairness for women, underrepresented minority students, and first-generation college students. One subscale, consisting of far fewer items than the legacy instruments, and no bias, will be cross-normed with the legacy instruments in order to provide a broad measurement of conceptual Newtonian mechanics and ensure wide adoption. The validation process will be documented so that it can lay the groundwork for the development of additional assessment scales determined to be of value by the physics community. The NSF IUSE: EDU program supports research and development projects to improve the effectiveness of STEM education for all students. Through its Engaged Student Learning track, the program supports the creation, exploration, and implementation of promising practices and tools. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.