CAREER: Scalable Learning and Models for Mapping Instructions to Actions

Cornell University, Ithaca NY

Investigators

Abstract

Robust language understanding has the potential to dramatically improve the quality and accessibility of autonomous systems operating in complex environments. Already today such systems are becoming increasingly common, including self-driving cars, drones, and robots surveying disaster areas. Natural language interfaces open new opportunities for non-expert users to control complex systems and increase the accessibility of current systems. However, existing methods are limited in expressivity and, more often than not, disappoint users. This Faculty Early Career Development Grant will fundamentally transform how this problem is addressed, and provide new avenues to build systems with robust natural language understanding and ability to improve and learn through interaction with users. The project's five-year goal of grounded language understanding directly connects to robotic agents and autonomous cars, and will enable new interdisciplinary applications and research directions. The goal of the research program is to create a new framework for mapping natural language instructions to actions. Instead of taking a modular approach, this work adopts a single-model view, where input text and raw visual observations are directly mapped to actions. While the approach includes components that can be trained and re-used separately, it does not require any intermediate symbolic representation, and does away with the need for different types of training data, as required to train conventional modular systems. The five-year goal of this project is a continuously learning reflective autonomous agent following natural language instructions in realistic environments. The research will address learning from sparse natural signals, reasoning about complex sequences of instructions, learning continuously from users, and developing interpretable models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.