NRI: Collaborative Research: Sketching Geometry and Physics Informed Inference for Mobile Robot Manipulation in Cluttered Scenes

The University Of Central Florida Board Of Trustees, Orlando FL

Investigators

Abstract

The goal of this project is to improve the ability of robots to manipulate and interact with objects, such as when assisting people to support their daily activities. The key idea is that people can provide robots with important information about their environment and the objects within their environment. In particular, people can use their cognitive skills to name objects, provide an understanding of the geometrical structure of objects, and describe an object's behavior in relation to other objects. Specifically, the project will develop a natural user interface that enables people to provide such information by drawing and sketching on top of the robot's view of the world. Physical simulation will then be used to fill in the missing gaps needed for a robot to complete autonomous manipulation tasks. Thus, the project aims to combine object sketching and physical simulation to better support mobile manipulation tasks as well as learn to perform new manipulation tasks when encountered. The project will support a "Put That There" task, where a user can simply give high-level manipulation commands, with the robot filling in the details necessary to complete the task in a cluttered environment. This project aims to improve goal-directed dexterous robotic manipulation in cluttered and unstructured environments through sketching and physical simulation. Robots operating in human environments face considerable uncertainty in perception due to physical contact and occlusions between objects. This project will address such perceptual uncertainty by combining methods for probabilistic inference with natural sketch-based interfaces to extract, label, and automatically infer the geometry, pose, and behavior of objects in complicated scenes. From a human usability perspective, the project addresses how to best create a sketching language and interfaces for intuitive human-in-the-loop extraction of object geometries and behavior from robot sensing. The planned exploration into sketching methods will also explore what underlying representations, raw point clouds, RGB images and video, or RGBD images will be most conducive to supporting accurate geometry extraction and grasp location identification. Given sketched objects, the project will develop probabilistic physically plausible methods for scene estimation that will enable perception for manipulation in cluttered environments. These methods build upon advances in physical simulation to constrain scene estimates to only plausible configurations to both improve estimation accuracy and enable computational tractability. The project will also develop a "Put That There" testbed using a tablet-based web application to support exploration of these concepts as well as act as user studies to evaluate geometry extraction accuracy and the robustness of physics-based scene estimation algorithms.