Perception-Driven Models for Granular Material Manipulation

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

This project seeks to provide robots with new capabilities to predict how granular materials, like sand or soil, will respond to physical actions, and to improve those predictions through on-going physical interaction, observation, and learning. First-principles computational models built up from the forces between large numbers of individual grains are too cumbersome for robot control. In this project, the granular material is compactly characterized as a graph, consisting of nodes representing aggregated regions of material, which are connected by edges representing dominant inter-region interactions. The graph representation is integrated with an artificial neural network, creating a powerful computational object called a graph neural network (GNN). The GNN can be used, for example, to generate a series of robot operations to form the granular material into a desired shape, or to infer that unexpected resistance to digging may be due to a buried root or rock. The GNN is updated based on camera images and the forces sensed as the robot works. The results of this project will be demonstrated on excavator robots, with the eventual goal of enabling robots that can reliably form structures such as pits, channels, and level foundations in varied terrains, including sandy, rocky, or root-embedded soil. The datasets, models, and software libraries developed in this work will be made publicly available to the robotics and terrain mechanics communities. Because physics-based simulations of granular materials are too computationally expensive to be used on board a robot, this project explores how machine learning approaches can be applied to predict robot-terrain interaction efficiently. Specifically, the project will investigate how a state-of-the-art machine learning model -- graph neural networks (GNNs) -- can be applied to model large terrains such as those present in construction sites. The investigators' approach will identify portions of large geometric maps into localized GNNs to predict fine-grained dynamics for small portions of the terrain, allowing GNNs to apply to much larger environments than was possible in the past. Moreover, because terrain behavior changes depending on the material characteristics and the presence of hidden objects, the project will develop multi-material GNNs to predict a range of terrain behavior, and these models will be used to help robots map out a terrain by feel in addition to vision. The broader impacts of this project include the involvement of undergraduates in research activities, outreach to underrepresented groups to broaden participation in computing, as well as the release of open-source software for modeling and manipulating granular materials. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.