CAREER: Manipulation of Novel Objects via Non-Smooth Implicit Learning

University Of Pennsylvania, Philadelphia PA

Investigators

Abstract

This Faculty Early Career Development (CAREER) project will create physics-inspired learning methods for data-efficient robotic manipulation of novel objects – that is, previously unseen objects about which the robot has no prior knowledge. The result will enable future generations of robots to provide meaningful assistance throughout the daily lives of human users. To achieve this, robots must be able to quickly learn about their surroundings through physical interactions, particularly in chaotic settings beyond carefully controlled laboratory conditions. This will require robots to gain new capabilities beyond the current state of the art. This project will provide robots with the ability to determine critical characteristics of surrounding objects -- despite variations in shape, size, color, and material -- such as whether they move when touched, whether they are soft or stiff, and whether they bend or twist. A robot should be able to enter a room for the first time, briefly investigate the objects in that room, and then safely accomplish an assigned task. For example, an in-home robot might maneuver about a kitchen, encountering new food items or culinary tools, and then interact with those items to help prepare a meal. This project will advance a range of life-improving robotic applications, including in-home assistive care, dangerous search and rescue operations, or small-scale manufacturing. To train and inspire the next generation of engineers, investigators, working with educators in the Philadelphia Public School District, will develop an educational unit leveraging a robotic simulator to demonstrate concepts from high-school algebra. This project brings together concepts from non-smooth dynamics, learning, and control, to enable robots that perform dexterous manipulation of previously unseen objects, using data gathered in real time from a cluttered environment. For example, a robot may interact with a set of novel objects for at most a few seconds or minutes, then precisely perform, with human-like dexterity, complex tasks such as tool use or in-hand manipulation. The need for this project is driven first by the dependence of functional robotics on interaction between the robot and its environment, which is notoriously difficult to model, and second, by the reliance on predictive models of both model-based and sim-to-real methods for control. This project addresses the modeling of discontinuous contact-driven dynamics by gathering all sources of non-smooth behavior into a set of contact forces. An implicit loss function, which itself uses convex optimization to estimate non-smooth contact forces, can be minimized using gradient-based methods to find a set of the smooth parameters that describe the physics of robot-object interactions. To provide data-efficient learning of robot-world interaction, this project explores the following three primary research thrusts: (1) development of foundational implicit-learning frameworks with physics-inspired structure for predicting robot-world interactions, (2) dynamic manipulation of novel rigid and soft objects by unifying tactile and visual sensing with motion prediction, and (3) combining these two learning systems with control and reinforcement learning for closed-loop performance. Together, these thrusts will provide new robotic capabilities when dealing with novel objects, across a range of manipulation tasks including in-hand manipulation and object reorientation, whole-body manipulation (using limbs, torso, and other body parts) to maneuver, push, and drag heavy objects, and multi-arm manipulation of large or unwieldy objects. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.