CAREER: Generative Item, Response, and Feedback Models in Assessment and Learning

University Of Massachusetts Amherst, Amherst MA

Investigators

Abstract

Personalized tutoring and feedback on performance or knowledge mastery, are two instructional strategies that have been shown to be effective at improving student learning outcomes. However, implementing these strategies, especially at scale, is costly in terms of the human resources required to provide them effectively. The use of Artificial Intelligence (AI) to provide students with feedback and personalized tutoring in digital learning platforms has the potential to reduce the human capital required to provide these services and to service growing numbers of learners effectively. This CAREER project will leverage generative language models (GLMs), a recent innovation in AI machine learning, to estimate learner knowledge levels and identify specific errors from open-ended learner responses. The resulting system will then be able to automatically generate personalized items and feedback, to support teachers and learners. Primarily grounded in middle-school math education with data collection and evaluation supported by ASSISTments and OpenStax, this CAREER project has the potential to benefit many teachers and learners. Other potential outcomes of his CAREER project include activities that expand the access of minority learners to real-world applications of AI and a new course on AI for education. This CAREER project includes three major research threads. First, the project team will develop a family of open-ended item response theory and knowledge tracing frameworks for open-ended math items. The key technical challenge will be to inject learner knowledge states to steer GLMs towards generating personalized response predictions according to each learner’s knowledge on different skills. These models will power teacher dashboard tools and learner error detection tools during tutoring activities. Second, the project team will develop GLM-based automated math item generation methods to meet the needs and interests of each learner and evaluate them in a randomized controlled trial. The key technical challenge will be to control the generated items according to human specifications on item context and both mathematical and language complexity. Third, the project team will develop a GLM-based automated feedback generation framework and explore its usage in both common wrong answer feedback and tutoring dialogue turn generation. The key technical challenge will be to learn how to leverage effective teacher-written feedback messages and use them as input examples for GLMs. The team will also explore learning-from-teacher-edit methods to constantly improve the quality of generated feedback over time. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.