NRI: Collaborative Research: Experiential Learning for Robots: From Physics to Actions to Tasks

Johns Hopkins University, Baltimore MD

Investigators

Abstract

Recent advances in machine learning coupled with unprecedented archives of labeled data are advancing machine perception at a remarkable rate. However, applying these advances to robotics has not advanced as quickly because learning for robotics requires both active interaction with the physical world, and the ability to generalize over a variety of task contexts. This project addresses this knowledge gap through the development of new learning methods to produce experience-based models of physics. In this approach, an object or category specific model of physics is learned directly from perceptual data rather than deploying general-purpose physical simulation methods. These physical models will support both direct control of action - for example pouring a liquid into a container, and the learning of the physical effects of sequences of actions - for example planning to handle fluids in a laboratory. More generally, these methods will provide a means for robots to learn how to handle fluids, soft materials, and other complex physical phenomena. The proposed experiential learning framework will build on recent advances in deep neural networks. The key problem is to learn the mappings between raw perceptual and control data via a low-dimensional implicit physics space representing a perception-based physical model of how an object acts in the environment. Three directions will be investigated: 1) the development of experiential physics models for object interaction and fluid flow that have strong predictive capabilities, 2) creating mappings directly from experiential models to control of actions such as pouring or moving an object, 3) the assembly of local experience-based controllers into complex tasks from interactive demonstration. Additionally, the project will develop unique data sets that include physical models, simulations, data components, and learned components that other groups can access and build on to enable comparative research similar to what has emerged in machine perception.