RI: Small: Understanding and Synthesizing People in 3D Scenes

Cornell University, Ithaca NY

Investigators

Abstract

This project advances fundamental research into understanding people in images, and how they interact with their environment. This goal is important because people are crucial in many real-world applications that involve imagery. For example, vehicles and robots should perceive people, predict their behavior, and operate safely around them. In the realm of computer graphics, algorithms that generate content for movies and games should depict people with realistic appearance and who interact with their environment in realistic ways. Developing a computational understanding of people sufficient for such applications will involve, crucially, understanding the connection between people and the scenes they inhabit. For instance, in a typical street scene, people will tend to appear walking on sidewalks or crosswalks -- but if it suddenly starts raining, they might look and act differently, for instance, by carrying umbrellas, hurrying for shelter, etc. This project seeks to build such a computational understanding of people in scenes via new human-centric methods for perceiving the world. The research will also be coupled with educational activities, including efforts to broaden participation in computing. The technical goals of the project fall into two main thrusts corresponding to a computer vision and a computer graphics goal: (1) human-centric understanding of images, and (2) realistic synthesis of people into images. Both of these goals will be driven by new machine learning methods that will answer the following series of questions: given an image of a scene, where might a person appear, what would they be doing, and how would they look? These questions are distinct from the typical computer vision task of identifying where humans actually are in an image, and instead involve reasoning about where they could be. This reasoning is highly dependent on the scene depicted in the image -- are there benches, is it raining, etc. -- and also on the 3D geometry of the scene -- e.g., whether a particular surface is horizontal or vertical. Hence, the project will explore scene- and 3D-aware learning of people and their interactions with the world. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.