RI: Small: Perceptually Grounded Learning of Instructional Language

University Of Texas At Austin, Austin TX

Investigators

Abstract

This project is developing methods that allow a computer to automatically learn to understand and generate instructions in human language. Traditional approaches to natural-language learning require linguistic experts to laboriously annotate large numbers of sentences with detailed information about their grammar and meaning. In this project, instructional language is initially learned by simply observing humans following instructions given by other humans. Once the system has learned reasonably well from observation, it also actively participates in the learning process by following human-given instructions itself, or giving its own instructions to humans and observing their behavior. The approach is being evaluated on its ability to interpret and generate English instructions for navigating in a virtual environment (e.g. "Go down the hall and turn left after you pass the chair."). A novel machine learning method infers a probable formal meaning for a sentence from the resulting actions performed by a human follower, and then existing language-learning methods are used to acquire a language interpreter and generator. The learned system is being evaluated in a range of virtual environments, testing its ability to follow human-provided natural language instructions to achieve prescribed goals, as well as to generate natural language instructions that humans can successfully follow to find specific destinations. The methods developed for this project will contribute to the development of virtual agents in games and educational simulations that learn to interpret and generate English instructions, and eventually aid the development of robots that can learn to interpret human language instruction from observation.