CAREER: Robot Perception of Human Physical Skills

Brown University, Providence RI

Investigators

Abstract

This award is funded in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Everyday human activities are impressive feats of physical intelligence - from careful placement of feet to avoid obstacles when walking, to the precise and highly coordinated movement of fingers to type a sentence. Robots with even a fraction of human physical intelligence could revolutionize lives by automating repetitive tasks. Despite advances however, robots with such physical abilities remain elusive. This project takes a step towards more capable robots by building 3D computer vision and machine learning algorithms for automatically analyzing human skills from large-scale image and video collections readily available on the internet or captured in the wild. It will produce a large repository of high-level physical skills that can then be transferred to robots. The education and outreach activities of the project will impart theoretical knowledge in robot perception and provide practical experience to graduates, undergraduates, and high school students. Furthermore, the project will lead to advances in computer vision-based understanding of human physical skills, in-the-wild capture of a significant amount of skills data and help to solve problems outside of CS such as in the study of the neuroscience of hand manipulation in monkeys. To meet the research goals, the project will advance the state of the art in computer vision-based modeling and estimation of human physical skills from large-scale visual data. Existing methods are limited to operating in structured environments and cannot capture interactions in unconstrained visual data taken in cluttered environments like homes. To address this limitation, the project will build (1) neural networks to model and estimate human physical properties such as shape and articulation from unconstrained data, (2) neural networks that model and estimate human motion and interaction from videos, and (3) methods for gathering and analyzing large amounts (10,000 person-hours) of unconstrained videos of human activities to build a repository of physical skills. This repository will inform the transfer of skills from humans to robots. The long-term aim of this research is to demonstrate that learning from images and videos is a viable path for robots to gain human-like physical abilities. To meet the education and outreach goals, the project will integrate theory and practice by acquiring several cameras and robot arms to teach an advanced course, a semester-long undergraduate research experience program, and a virtual workshop program. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.