NRI: Robots that Learn to Communicate through Natural Human Dialog

University Of Texas At Austin, Austin TX

Investigators

Abstract

Robots are increasingly capable and are on the threshold of becoming a ubiquitous technology. For robots to be truly useful, people must be able to effectively communicate their needs in everyday human language. Although there is a growing body of research on natural-language processing for human-robot interaction, it typically requires some form of explicit supervision provided by an engineering expert and involves unnatural, laborious training to obtain robustness and coverage. This project involves the development of human-robot dialog systems that learn to communicate with users through natural dialog, learning from repeated normal user interactions to become more robust and capable. The project supports the education of students in the areas of natural-language processing, human-robot interaction, and machine learning, where there is significant demand for educated personnel. It is integrated with the university's Freshman Research Initiative, which gets undergraduate students involved in research in their first year. In order to develop human-robot dialog systems that learn to improve their communication skills through normal user interactions, the project integrates and adapts learning techniques from three currently disparate technical areas: semantic parsing, spoken dialog management, and perceptual language grounding. The project adapts and integrates techniques for semantic-parser learning using combinatory categorial grammar (CCG), dialog management using Partially Observable Markov Decision Processes (POMDPs), and multi-modal language grounding using both visual and haptic sensors, in order to develop a dialog system for communicating with robots that comprise the Building Wide Intelligence (BWI) system being developed at the University of Texas at Austin. The research integrates the PI's expertise in semantic parsing and language grounding with the co-PI's expertise in robotics and reinforcement learning, forming a unique interdisciplinary team for developing novel and effective systems for human-robot interaction. The project includes rigorous evaluations using controlled experiments on a range of tasks using both on-line simulations with crowdsourced users, and natural user interaction with a mobile robot platform consisting of a wheeled Segway base and a Kinova robot arm being developed for the BWI system.