Accelerating Risk-of-Bias (RoB) Assessment in Environmental Health Studies Using Large Language Models

Pico Portal, Inc., Saint Petersburg FL

Investigators

Abstract

Grant No: 1R43ES037589-01 PI Name : Agai, Eitan Project Summary PICO Portal, Inc. proposes the development and rigorous evaluation of advanced AI-driven tools to automate, increase efficiency, and reduce the costs of conducting Risk-of-Bias (RoB) assessments in environmental health systematic reviews (SRs). SRs are the "Gold Standard" for synthesizing evidence to inform decision-making in environmental health, healthcare, health policy, and social science. However, these reviews are resource-intensive, requiring significant investments of time, expertise, and financial resources, particularly for tasks such as RoB assessments and data extraction. This project aims to utilize the capabilities of large language models (LLMs), such as Meta AI's LLaMA, OpenAI’s GPT, and Anthropic’s Claude, to address these challenges. Specifically, we will focus on developing, fine-tuning, and validating LLM-based workflows to accurately identify and extract critical text passages from studies related to per- and polyfluoroalkyl substances (PFAS). We aim to refine the model through iterative testing and optimization until its performance closely mirrors that of human experts. Achieving a high level of agreement between AI-extracted passages and those identified by trained professionals is crucial to ensuring that the model is not only accurate but also reliable and effective in real-world applications. The successful implementation of this project will increase efficiency and reduce costs for conducting SRs in environmental health. Furthermore, this innovation could extend beyond environmental health to other fields such as healthcare, health policy, and social science. It could also benefit any review process, including scoping the literature, identifying research gaps for funding agencies, and conducting competitive intelligence for the private sector.