Defining Almost Correct: Quantifying Student Understanding Hidden in Wrong Answers

Rowan University, Glassboro NJ

Investigators

Abstract

This project aims to transform the way faculty assess student learning in undergraduate physics courses. Quantitative assessments of student learning in physics have generally focused on whether a student got the "right" answer on a multiple-choice exam. This kind of analysis fails to capture how close a student's understanding is to being "right." It therefore cannot track whether or how students' understanding improves or progresses. This project will develop sophisticated scoring methods for multiple-choice tests that can reveal students' productive, but "wrong" ideas. This method should result in better-informed and more equitable decisions regarding instructional practices. For instance, this kind of analysis could determine if students have improved their understanding after instruction, even if they still do not get the right answer. Such information could be particularly helpful with less-prepared students who have further to go in their learning process. Since a disproportionate number of these students are from groups that are under-represented in physics, using a simple right/wrong analysis treats these groups inequitably. More fully representing student learning is vitally important for making better decisions regarding instructional practices. This project will use two kinds of analysis to determine which wrong answers from a set of choices are better than others. The first process is a statistical analysis that determines the most common sequence of wrong answers students traverse before arriving at the correct answer. The second process involves qualitative interviews to uncover student thinking about the reasons for their choices. Better understanding the progression of student learning across multiple disciplines should help develop a larger and more inclusive STEM work force to meet the growing needs of the U.S. economy. This project aims to meet the critical need for a more complete scoring metric for research-based assessment instruments, by developing a new assessment tool to measure student learning in novel ways. The new approach is based on data from a commonly used research-based assessment instruments in physics, the Force and Motion Conceptual Evaluation. This project will begin by developing a ranking of incorrect responses to each question on the Force and Motion Conceptual Evaluation, based on quantitative analyses of student responses. Next, it will reconcile rankings from multiple analyses to generate a unified ranking for each question. From this unified ranking, the project will define a metric to represent overall student knowledge. Then, the project will develop a user-friendly assessment tool (software) to analyze student response data and calculate the new learning metric. Next, by applying the new assessment tool to existing data, the project will identify patterns in student response progressions that differ based on instructional factors or student demographics. Finally, the project will connect the unified rankings to the learning progressions literature by interviewing students about why they choose various responses. The outcome will be a new assessment tool that should allow researchers and instructors to more deeply analyze data about their students' learning. This knowledge could lead to more informed decisions regarding instructional choices. The results of this project may be applied to any multiple-choice, research-based assessment instruments in any discipline. Thus, the project has the potential to significantly improve the ways in which data about learning are interpreted in many different contexts. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.