S&AS: FND: Context-Aware Active Data Gathering for Complex Outdoor Environments

University Of Minnesota-Twin Cities, Minneapolis MN

Investigators

Abstract

Traditional agents are programmed to acquire information by recognizing and attending to predetermined areas and targets in a given environment. Recent advances in deep learning models and miniature hardware platforms are providing artificial agents unprecedented capability in processing and interpreting visual data. These advancements create an exciting opportunity to build intelligent machines running with greater autonomy and adaptability. Toward this goal, this project investigates new methods that enable multiple unmanned aerial systems to understand and explore complex outdoor environment by actively seeking, acquiring, integrating, and processing visual information across space and time. The developed framework with enhanced adaptability, self-awareness, and generalizability will be applicable to autonomous systems in broad applications such as environmental monitoring, search and rescue, self-driving cars, smart health, and manufacturing domains. Throughout the project, the principal investigators will make project results including created datasets, trained models, code, and papers publicly available. The new integrative research combining vision, planning and actuation will be incorporated into teaching materials, underrepresented and undergraduate research projects, as well as K-12 outreach activities. The project seeks to develop algorithms for context-aware active sensing which also incorporate energy constraints. This will be achieved by: First, proposing new deep learning models for holistic attention prediction with multiple aerial views. The models will leverage external knowledge to enable inference and generalization in unseen contexts. Second, by developing new view and path planning methods that are efficient and aware of systems' energy, mobility and sensing constraints. Third, by contributing novel online learning methods that adapt based on uncertainty to implement adaptiveness and awareness to changing environment. Experiments to validate the findings will take place both indoors in the newly-renovated Shepherd UAV Lab at the University of Minnesota and in the field at the Cedar Creek Ecosystem Reserve. The results of this project have the potential to inspire further research into intelligent and integrative perceptual, planning and actuation systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.