EAGER: Robots that Learn to Communicate with Humans Tthrough Natural Dialog

University Of Texas At Austin, Austin TX

Investigators

Abstract

This EArly Grant for Exploratory Research explores the possibility of developing more user-friendly and capable robots that learn to understand commands in natural human language. The experimental system developed aims to engage users in natural conversation, clarifying linguistic instructions that cannot be understood, and learning from this interaction to more robustly interpret future commands. This fundamentally new approach is hypothesized to overcome limitations of more-costly previous approaches that require either direct programming or detailed annotation of per-assembled linguistic data, and still frequently fail to cover issues that arise in real user interactions. The resulting exploratory prototype is evaluated on real interactions with human users, experimentally testing its ability to improve its accuracy and flexibility at interpreting human instructions over time, through normal everyday use. This novel approach aims to improve human interaction with intelligent multi-robot systems that aid the residents and visitors of a large, multi-use building. This fundamental research also supports computer-science education in the growing areas of natural-language processing, human-robot interaction, and machine learning, where there is significant national demand for knowledgeable personnel. The technical approach explored is a novel integration of learning techniques from three currently disparate areas: semantic parsing, spoken dialog management, and perceptual language grounding. Semantic parsing is the task of mapping natural language to a formal computer-interpretable language using compositional semantics based on syntactic linguistic structure. Dialog management concerns controlling multi-turn natural language interaction to aid comprehension and task completion. Perceptual grounding concerns associating words and phrases in language to objects, properties and relations in the world as perceived by the robot's sensors. Although there has been recent significant progress in each of these individual areas, no one has previously explored integrating them to support learning for human-robot communication through natural dialog. This exploratory research adapts and integrates techniques for semantic-parser learning using combinatory categorial grammar, dialog management using Partially Observable Markov Decision Processes, and multi-modal language grounding using both visual and haptic sensors, in order to develop a novel dialog system for communicating with robots that comprise the innovative Building Wide Intelligence system being developed at the University of Texas at Austin. The exploratory methods are evaluated using controlled experiments on a range of tasks using both on-line simulations and crowdsourced users, and natural user interaction with a mobile robot platform consisting of a wheeled Segway base and a Kinova robot arm.