RI: Small: Scaling up Robot Learning by Understanding Internet Videos

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

Even simple common sense knowledge (for example, that drawers can be opened by pulling on handles, or how to efficiently find the way to a coffee shop in a new hotel without a map) is hard to incorporate into control algorithms in an automated manner at scale. Incorporating such knowledge in the form of hand-designed rules leads to brittle systems and does not scale. This necessitates the use of machine learning to automatically learn such knowledge from data. Current machine learning techniques largely learn by discovering this knowledge by themselves via trial-and-eror. Not only is this computationally expensive, but it also results in specialized behavior that does not generalize to new operating conditions. This makes it challenging and tedious to deploy such learned control algorithms. At the same time, such world knowledge is readily depicted in datasets of first and third-person videos of people conducting different tasks found on the Internet. This project will advance the state-of-the-art by developing techniques to extract knowledge from such videos, to aid learning of decision making and control algorithms. Techniques developed in this project will enable easier, faster, and better training of robots (such as in automated manufacturing). This will enable broader adoption of learned policies for basic navigation and manipulation tasks in previously unseen environments. Effective policies for basic robotic tasks will aid future robotics research on higher-level problems (such as task planning, and human-robot interaction), and computer vision research on interactive problems (like active learning and active perception). The project will also contribute to the education of graduate and undergraduate students by the development of specialized courses and involvement in research, and the research community at large through accessible dissemination of research. In order to learn robot policies from Internet videos, the project will develop a framework that leverages the synergies between learning via direct interaction and large-scale video understanding, to learn from and for each other. Researchers will develop video understanding techniques that allow a) building representations that are sensitive to object states, b) acquiring skills for short-range navigation and manipulation, and c) learning value functions that encode world knowledge for task completion. Collectively, these will enable sample efficient learning of control policies that generalize well. Researchers will collect and curate relevant datasets. These datasets will be processed to make them amenable for learning useful policies and representations, by aligning relevant videos in space and time, and grounding transitions into actions. Useful representations for policy learning will be extracted by learning state-sensitive features, parameterized skills, and value functions that capture knowledge about the world. The effectiveness of the proposed framework will be demonstrated through faster learning and better generalization of learned behaviors as compared to existing approaches. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.