CHS: Small: Integrative Wide-Area Augmented Reality Scene Modeling

University Of California-Santa Barbara, Santa Barbara CA

Investigators

Abstract

Augmented reality (AR) is a technology that superimposes computer-generated visual, auditory, or other sensory information onto the physical world, so that they appear to be part of the actual environment. The goal of the proposed project is to develop and evaluate new methods for creating, maintaining, and improving large-scale scene models to enable wide-area AR. For example, consider an AR application for firefighters, which superimposes predicted changes in fire perimeter and coordinated action plans to help them understand weather impact and communicate the current plan of attack. Augmented reality is characterized by the fact that these visualizations are three-dimensionally registered with the real world, keeping them matched to the physical environment in real-time even as the user moves around their environment. In order to place new virtual annotations (and to keep track of previously placed annotations), a three dimensional "scene model" that represents the physical world locations needs to be created. Ideally, this would be done on a global, world-wide scale to include all possible places where AR experiences are possible or have even already occurred. Since it is difficult for one person or organization to collect the data needed for such a wide-area scene, the developed system will aggregate crowd-sourced data of different modalities (e.g., images, videos, and 3D geometrical meshes) from multiple sources, leveraging semantic information in addition to basic image and point cloud data. By providing capabilities for remotely guiding a local AR user to capture new imagery or sensor data to achieve more accurate or complete models, crowd-sourced modeling can be directed over time to create and continuously update scene models of large-scale areas, such as a university campus or even a city. The research will result in a system that effectively integrates multiple components to provide new opportunities to remotely navigate, explore, and augment physical spaces for AR applications in government, education, industry, and consumer spaces. To accomplish the above objectives, the researchers will implement and utilize a Dynamic Hybrid Scene Model Server that accepts crowd-sourced image, video, and point cloud data, and continuously performs smart data integration and completion, leveraging machine learning approaches to infer semantic information from the raw image and point cloud data and to fill in missing information. The main challenge here will be to design the learning approaches in such a way that it will not require access to all the low-level data fusion and filtering components, simply because the information may not be available in the crowd-sourced data. Augmented Reality and Virtual Reality user interfaces will be developed and evaluated to deal with imperfect and incomplete environment models produced by the server, allowing remote users to virtually navigate through modeled spaces and to provide guidance to the local AR users. On the human interface side, the project focuses on research in remotely navigating and exploring "visual reality", virtual models of the real-world spaces created from the crowd-sourced imagery and creating content for augmenting the visual reality. The proposed methods will address limitations in current mixed reality applications in modeling and remote navigation, providing users an experience of remote visual reality that is valuable as a replacement for physical navigation, as a training aid for planned activity, and as a way to share information in augmented and virtual reality environments. The project will leverage existing work by the team of researchers and others on image-based modeling, virtual scene navigation, and virtual annotation for remote collaboration, and it focuses on both the key system components and the user experience to explicitly support navigation and augmentation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.