EAGER: Shared Visual Common Ground in Human-Robot Interaction for Small Unmanned Aerial Systems

Texas A&M Engineering Experiment Station, College Station TX

Investigators

Abstract

This project will create a computational theory of visual common ground, allowing users to give directives to a robot (or other team members) and receive confirmation or constraints through visual communication over a shared visual display. The motivating example is an urban search and rescue (US&R) professional tapping, sketching, and annotating on an iPad in order to direct a small unmanned aerial system (sUAS) without training. Previous work in human-robot interaction with common ground has been limited to natural language, but recent work has shown that having all team members see the robot's eye view in unmanned ground robots significantly improved performance and situation awareness. The proposed work populate the computational theory using the Shared Roles Model to represent the inputs (directives, notations), outputs (display viewpoint, form, size, location, content, etc.), and transformations (visual communication engine). The computational theory will be prototyped, refined, and tested by US&R practitioners flying realistic sUAS missions at Texas A&M's Disaster City. Intellectual merit: The project will create a computational theory of visual common ground that will enable two-way human-robot interaction using visual communication mechanisms such as tapping, sketching, and annotation on shared visual displays on mobile devices such as iPads, smartphones, and tablet PCs. The results will advance the fields of human-robot interaction, artificial intelligence, and cognitive science. Broader impacts: The results could revolutionize how people use mobile devices to interact with robots (and with each other) using naturalistic visual mechanisms, bypassing extensive training. The project will actively recruit women, Hispanics, and persons with disabilities to participate through REU programs. An open source visual communication toolkit for HRI researchers will be produced. The results will improve robots for public safety, remote medicine, and telecommuting, and could also immediately help save lives through incorporation into Texas Task Force 1.