FairFL-MC: A Metacognitive Calibration Intervention Powered by Fair and Private Machine Learning

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

Students often have difficulty estimating their own level of knowledge. The goal of this project is to research ways to improve students' ability to estimate their knowledge, using a student support system consisting of short training exercises that will be personalized with artificial intelligence (AI) methods. While there is abundant research on AI methods in educational contexts, such projects rarely consider some of the key social and human factors, such as privacy and fairness, that are needed for widespread adoption of personalized educational software. This project addresses these issues with a novel decentralized AI framework that is specifically for education contexts. The project framework will enable researchers to create AI systems that provide feedback to students as part of their training exercises, all without directly accessing their data and while also training the AI system to reduce biases related to key aspects of students' identity, such as their demographics. The training exercises will include educational activities where students estimate their test scores, receive feedback from the AI system, and reflect on their knowledge. The privacy and fairness capabilities of the project framework will transform postsecondary online learning, which is poised to benefit from emerging AI-driven learning technologies but has yet to fully realize these benefits. The project will directly benefit students participating in the research as they will improve their knowledge estimation skills, prepare more effectively for tests in class, and learn about potential privacy violations and AI biases. Given the fairness focus of the project, the team of researchers will pay special attention to benefits for students from groups traditionally underrepresented in STEM (Science, Technology, Engineering, and Mathematics), ensuring that the AI-powered framework is equally helpful for them and that their perspectives on privacy and fairness receive special attention. This project will advance AI research by incorporating, both, a strict privacy guarantee for student data and fairness considerations across multiple student demographic groups. Additionally, it will advance education research by determining how effective preemptive feedback is for improving knowledge estimation skills, and will examine the mechanism by which preemptively improving knowledge estimation influences academic outcomes. In particular, the project will achieve four research objectives through interdisciplinary innovations in both learning sciences and technology. First, the team will determine how much students' metacognitive calibration can be improved via AI-powered preemptive feedback, which may be perceived differently by students than post hoc feedback. Second, the project will expand theoretical understanding of metacognitive calibration and calibration interventions by studying the mechanism by which the intervention in the project works. Third, the team will address the fundamental tradeoff between the fairness and accuracy of AI models via an innovative federated learning model. Fourth, the team will evaluate the AI framework on real-world education datasets and compare its performance with the state-of-the-art baselines in terms of protecting privacy and mitigating bias. The project team will disseminate results of the project through workshops, publications, and interactive activities, and will train undergraduate and graduate students from diverse backgrounds throughout the project. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.