CAREER: Minds and Machines: Exploring Engineering Faculty Member Mental Models of Generative AI and Instructional Decisions

Virginia Polytechnic Institute And State University, Blacksburg VA

Investigators

Abstract

This CAREER project will explore how engineering faculty members perceive and integrate generative artificial intelligence (GAI) technologies into their teaching practices, particularly regarding assessments. Generative AI models are rapidly transforming many sectors in society, including education. As these GAI models become more capable of performing tasks typically required in engineering classrooms and assignments, it is crucial for educators to adapt their teaching and assessment strategies. Those adaptations can ensure that engineering students appropriately learn fundamental concepts in their fields while also learning to work effectively alongside these advanced tools. To realize that vision, it is important to understand how faculty members view these technologies and make informed assessment decisions. Based on that motivation, this project addresses the pressing need to initially characterize and then improve faculty members’ mental models about GAI. In turn, more realistic mental models can help faculty members make better informed decisions about how to adapt their assessment approaches and integrate these technologies into engineering curricula. Through research and education activities the project will help engineering faculty members develop their mental models and assessment practices while also offering the additional benefit of supporting broader efforts to prepare future engineers who are adept at leveraging GAI in their professional lives. This project aims to understand and improve the mental models of GAI held by engineering faculty members and how these mental models influence their instructional decisions, particularly in assessment strategies. The project is grounded in the mental models approach from risk communication and system dynamics to understand perspectives of these technological systems along with the theory of planned behavior as a lens toward understanding faculty member intentions and behaviors. Together, the comprehensive framework will provide a conduit for examining the cognitive and social factors shaping faculty members’ intentions and behaviors related to adapting their assessment strategies in response to GAI model capabilities and availability. The research activities of the project will employ an exploratory sequential mixed-methods design. The initial phase will involve in-depth, semi-structured interviews with two groups of participants: engineering faculty members across multiple disciplines and GAI experts. These interviews will explore current mental models and assessment adaptation strategies. Qualitative findings from phase one will inform the design of an annual cross-sectional trend survey administered to a nationally representative sample of engineering faculty throughout phase two. Activities in this quantitative second phase will support inferential claims about associations between faculty and institutional characteristics and mental models. Likewise, large-scale qualitative data analysis of open-ended items on these annual surveys will support investigations into systematic associations between groups, mental models, and assessment decisions. Administering the survey at multiple time points will also enable identification of how mental models and practices evolve as the relevant technologies continue to change. The expected outcomes from this research include a nuanced understanding of faculty attitudes towards GAI and assessment, development of tailored faculty development initiatives, and creation of educational resources to support effective GAI integration into engineering education. Building on those findings, education activities associated with the project will involve faculty workshops and communities of practice to support informed engagement with these technologies and appropriate assessment adaptations. Partnerships with multiple universities and professional organizations will ensure broad dissemination and impact, contributing to a more informed and prepared engineering faculty and, ultimately, a workforce equipped to harness the potential of GAI. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.