III: Medium: Human-in-the-loop Visual AI for Precise Scientific Measurement

University Of Massachusetts Amherst, Amherst MA

Investigators

Abstract

Artificial intelligence (AI) is revolutionizing scientific discovery by enabling researchers to analyze large and complex datasets that are otherwise beyond the reach of traditional methods. However, AI is not without limitations—it can introduce systematic errors, make unacceptable mistakes, and often lacks the statistical guarantees that scientists require. This project aims to unlock the full potential of AI in scientific research by developing techniques that integrate imperfect AI models with human-in-the-loop feedback and rigorous statistical estimation. These methods will enable high-throughput, high-precision scientific measurements with quantified uncertainty from large-scale datasets. The work will be grounded in four high-impact applications spanning environmental science and sensing—using data from satellite imagery, radar, acoustics, and sonar. By systematically evaluating the approach in real-world settings, this project will help expand scientists’ ability to use AI responsibly and effectively for discovery and decision-making in areas such as environmental monitoring and disaster response. The project will support interdisciplinary training for PhD students and undergraduate researchers by incorporating its research themes into undergraduate and graduate courses, designing course projects that address real-world challenges in deploying AI, and organizing community workshops that amplify the impact of AI in scientific domains. The project will develop a new framework called active measurement, which combines imperfect AI predictions with targeted human feedback to produce unbiased, statistically grounded estimates of scientific quantities. Active measurement integrates Monte Carlo estimation of a scientific quantity into the full life-cycle of interactive AI model development. Unlike previous approaches, active measurement is fully adaptive: it can dynamically select samples for humans to label and update the underlying models when resources are available. The research will investigate new methods for improving computer vision models and adapting them to novel domains using limited human input, including (1) actively constructing validation sets to improve domain adaptation, and (2) a general-purpose vision model that can be efficiently adapted to specific measurement tasks using examples and text-based guidance. The approach will be deployed and evaluated in collaboration with experts from academia, government agencies, and NGOs, across four domain applications: ecological monitoring, habitat mapping, acoustic wildlife detection, and disaster impact assessment. These efforts will validate the statistical rigor, scalability, and generalizability of the methods for scientific and policy-relevant use cases. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.