POSE: Phase I: Towards an Open-Source Ecosystem for Trustworthy Medical Foundation Models

University Of North Carolina At Chapel Hill, Chapel Hill NC

Investigators

Abstract

This Pathways to Enable Open-Source Ecosystems (POSE) project focuses on building a collaborative and transparent ecosystem to strengthen the reliability of artificial intelligence (AI) applications in healthcare. As AI tools increasingly support clinical decision-making, advancing their safety, robustness, and interpretability becomes critical. This project creates a national open-source community centered on the evaluation and continuous improvement of medical AI systems. By developing shared infrastructure, governance structures, and community engagement strategies, the project empowers researchers, clinicians, and technologists to collaboratively enhance these systems. The system-wide benefits include more consistent clinical decision-making, improved patient safety, and greater transparency in AI-assisted healthcare workflows. The project also seeks to educate a wide range of stakeholders on best practices for developing and deploying reliable AI, thereby strengthening the nation's leadership in responsible medical innovation. Healthcare providers and patients stand to benefit most directly from the outcomes of this initiative. The project will also provide educational resources and technical guidance on evaluating and applying reliable AI systems in practice. The outcomes of this initiative are designed to support a wide range of users involved in the evaluation, deployment, and oversight of medical AI systems. This POSE project develops the technical and organizational foundations necessary to sustain an open-source ecosystem focused on evaluating the trustworthiness of medical AI systems. Building upon an existing benchmarking framework, the project enhances its modularity, scalability, and security. The team will integrate additional evaluation dimensions, such as interpretability and develop tools to facilitate ongoing community-driven improvements. A transparent governance framework, including a steering committee and safety operating guidelines, will guide ecosystem development. Through workshops, comprehensive documentation, and collaborative events, the project will engage stakeholders from AI research, clinical practice, and open-source ecosystem design to co-develop evaluation protocols and tools. By enabling reliable assessment of medical AI systems, this project aims to advance scientific understanding of reliable machine learning and support the safe and effective integration of AI into healthcare. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.