RI: Medium: Robots That Learn From Description Through Synthesis and Analysis

Johns Hopkins University, Baltimore MD

Investigators

Abstract

Many types of physical work are performed most efficiently in collaboration. For example, garage mechanics, electricians, surgeons, and carpenters often employ an assistant or an apprentice to help them perform their jobs. The goal of the proposed project is to develop the underlying principles that would allow an intelligent robotic system to naturally collaborate with a human partner in such tasks. In particular, the proposed research studies the problem of performing an action in response to a complex command; for example, "When I hold these two pieces together with the holes aligned, place a 3 inch screw through the hole." The results of this research will be demonstrated in a system that is able to assist in typical assembly or cooking tasks. The research explores the development of perception-based classifiers that are composed, on demand, from the structure of the command itself. This classifier is constructed from pre-trained generic components and is then fine-tuned using simulation-generated data also derived from the structure of the command. This poses several challenging technical problems. The first problem is to create general-purpose perceptual components that correspond to the language units of the query's objects, actions, relationships, and activities. The second problem is to compose these units to form a classifier and to create methods for on-demand fine-tuning of the classifier from simulation data. The system must then be able to command a robot to perform the correct action in response to the command. Measures of success of this research will be 1) the breadth of capabilities the system is able to provide; 2) the specificity it is able to achieve when applied to representative video data; and 3) its ability to naturally interact with a human user to perform a collaborative manipulation task. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.