EAGER: Artificial Intelligence to Understand Engineering Cultural Norms

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

The rapid adoption of artificial intelligence (AI) tools across industries demonstrates that these tools will be used to generate content about engineering education and careers, in addition to being used to interact with students. AI tools like ChatGPT have been adapted to several industries, from real estate to journalism, and related AI technologies are predicted to have a profound impact on nearly every industry. This unprecedented adoption brings a myriad of concerns, including the impact of using AI tools to replace and thus devalue human interactions. This project will explore how AI text generation tools like ChatGPT portray stress and mental health in engineering. By examining how engineering is portrayed and comparing results to previously recorded interviews, the project will identify how these AI tools can be leveraged for qualitative research studies. Understanding engineering cultural norms and expectations, particularly around stress and mental health, is critical in addressing longstanding challenges in engineering related to chronic underrepresentation of groups and low student retention. The project is a fit with the EAGER program due to its exploratory nature, as AI text generation tools have not been used for data generation, despite their increasing use in data analysis. Further, the project is potentially transformational with its potential to introduce new research methodologies. Finally, the project will provide training and resources to the larger engineering education community to use AI tools. This project will explore the use of large language models (LLMs) to generate qualitative datasets. The project is exploratory in its goal to use LLMs as a research tool to understand engineering cultural norms related to stress and mental health and will be informed by the Engineering Culture framework. Results generated from LLMs will be compared with themes derived from qualitative datasets generated from student interviews about engineering culture. With the rapidly increasing availability of open source LLMs, researchers have, and will increasingly continue, to use LLMs and other AI tools for research. This has created a critical and urgent need to determine best practices to yield high quality data and analyses. To address this, our project asks the overall research question: How can LLMs be used to generate data for qualitative engineering education research studying engineering culture? The study will develop guidelines for how LLMs can be leveraged to produce quality datasets for cultural studies that enrich existing qualitative methods and overcome limitations of these methods. The project will specifically examine cultural understandings of mental health in engineering culture and compare LLM-generated datasets to student interviews about stress and mental health in engineering culture. Research to understand the knowledge distilled by LLMs will contribute to research on understanding perceived norms and assumptions about stress and mental health in engineering education and careers. Leveraging LLMs in qualitative research has the potential to enable new and synergistic methods for qualitative research. The project will develop resources for the engineering education research community to use AI in qualitative research. Ultimately, the proposed work will lay the groundwork to understand how LLMs can be best used in qualitative EER and engage engineering education researchers in the AI Revolution. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.