CAREER: Domain-Specific Data Capture on Ubiquitous Devices

Cornell University, Ithaca NY

Investigators

Abstract

Recent years have seen incredible progress in Artificial Intelligence (AI), largely thanks to improvements in the training and use of deep models. However, much of the progress in AI has come from training on massive publicly available datasets, such as, all of the images and text found on the Internet. By contrast, many important real-world problems depend on scarce or highly specific data that still must be captured manually. For example, consider the task of monitoring a crack in the foundation of an aging bridge, where the goal is to predict whether and when repairs should be made to ensure the safety of travelers. Or the case of monitoring a patient's wound outside of hospital settings. Solutions to these problems cannot be found on the internet; they require specific knowledge about a specific subject at a specific time. This project focuses on developing mobile applications to help regular users capture information that experts - e.g., doctors, scientists, and engineers - need to make important decisions. The project will also integrate research with education and outreach to high school students and teachers. This research focuses on the development of applications for domain-specific data capture on ubiquitous devices. The project will combine novel interaction design with adaptive tracking and registration strategies to build systems that help users capture target distributions of field data important to different downstream tasks. These systems will include tools that make it easy for experts to define target distributions of important observations, which can then be used to guide the collection of field data by non-experts using custom mobile applications. These applications will also allow the capturing users to update and adapt target distributions according to evolving field conditions. The project will further leverage guided capture systems to enable new types of visualization and analysis and develop tools to efficiently label captured data. The research integrates three key components into the design of these novel systems: (1) mobile tracking and registration; (2) reconstruction, visualization, and analysis; and (3) user interaction design and real-time guidance through Augmented Reality (AR). The integration of these components will help enable new applications. For example, new mobile tracking and registration abilities make it possible to register mobile devices against more complex target distributions, allowing us to offer more precise guidance for a broader range of downstream tasks. This integration makes the work highly interdisciplinary, with contributions in computer vision, computer graphics, and human-computer interaction, as well as collaboration with domain experts in downstream application domains. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.