EAGER: Improved Situation Awareness of Unknown Environments through a Robotic Augmented Reality Virtual Window

Mississippi State University, Mississippi State MS

Investigators

Abstract

When responding to a search warrant or 911 call, teams of police officers often have to enter and search a room. The problem the police officers face is that the contents of the room are unknown and might be dangerous. Therefore, the activity is stressful, and sometimes results in adverse outcomes. This project addresses this problem by developing a robot that police officers can send into a room before they enter. The robot uses cameras and other sensors to see the room, even in the dark. The police officers wear augmented reality headsets that allow them to see the room through the robot's cameras. In addition, the combination of augmented reality and the robot allows the police to see the room as though they had x-ray vision, giving them a virtual window through an otherwise solid wall. Seeing the room through this virtual window would make searching the room more efficient, reduce uncertainty and stress, and result in avoided injuries and saved lives. More broadly, the project will further the science of using robots and augmented reality with skilled teams performing dangerous work. The lessons learned could advance efforts to incorporate robots into related activities, including additional policing tasks, firefighting, inspecting buildings and warehouses, responding to alarm activation, searching and rescuing, and military operations. The objectives of this project are (1) to explore the use of an augmented reality virtual window to provide x-ray vision, where otherwise invisible objects can be seen, and (2) to use a robot to locate and identify objects in unknown, potentially dangerous, and stressful environments. These objectives are pursued in the context of a common task -- the room-clearing scenario -- undertaken by teams of police officers. This project will develop a non-weaponized robot, which team members will send into a simulated apartment with several rooms. The robot will view each room from robot-mounted cameras and sensors, and will present the view in an augmented reality (AR) virtual window. The cameras and sensors will see the room in multiple spectra, including visible, thermal, and laser, which will allow viewing in varying levels of illumination, including darkness. The project will leverage the investigators' longstanding relationship with regional police SWAT teams (Special Weapons And Tactics), who have agreed to provide personnel to be subject matter experts, design partners, and evaluation subjects. The project will be conducted in an existing tactical robotic testbed facility, and spatial understanding and situation awareness will be measured, using both standard and custom methods and surveys. The work has the potential to advance the fields of augmented reality, robotics, and human-robot interaction, as well as to advance the day when robots can be successfully integrated into SWAT teams. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.