CAREER: Improving Multi-Fingered Manipulation by Unifying Learning and Planning

University Of Utah, Salt Lake City UT

Investigators

Abstract

For robots to act autonomously as assistants in one's daily life, as surrogates for humans in dangerous environments, or as workers in busy factories and processing centers, they must be able to fluently grasp and manipulate objects that they have never previously encountered. This project investigates new approaches to perform grasping and in-hand manipulation using multi-fingered robot hands. The primary novelty in this work comes from examining new ways of combining knowledge gained automatically by the robot from its own sensors with models given to the robot by its programmer. The research team will perform extensive empirical evaluation of the algorithms developed in order to quantify the benefit of these new techniques over previously proposed analytic or data-driven approaches. This will allow for fast translation of the research results into robots used in industrial and service settings. This project will conduct educational outreach activities using robot manipulation to increase and reinforce middle and high schoolers' interest in computing. For autonomous robots to operate seamlessly as assistants in human environments, they must be endowed with the ability to aptly perform dexterous manipulation. Contemporary robot hardware provides the necessary dexterity and sensitivity to perform dexterous manipulation, but algorithmic shortcomings currently cripple deployment of robust multi-fingered grasping, regrasping, and in-hand manipulation of unknown and partially modeled objects. The research goal of this project is to unify concepts from model-based planning and data-driven learning for manipulation to improve dexterous manipulation of unknown objects. The first research thrust examines adding model-based constraints to perform grasp synthesis with a learned deep network. The second research thrust evaluates the hypothesis that learning feedback policies, approximate models, and graph structures from tactile and visual sensing will improve execution of pre-planned in-hand manipulation trajectories. The investigator proposes viewing these problems as an instance of probabilistic inference. This enables the robot to directly reason over uncertain sensory observations and partial object-pose and contact-state information. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.