CAREER: Toward Video2Sim: Turning Real World Videos into Simulations

Princeton University, Princeton NJ

Investigators

Abstract

This project develops new technology toward Video2Sim: automatically converting a video into a virtual world, where scenes are reconstructed, actions are re-enacted, and alternative outcomes are simulated by a computer. Such a system does not yet exist due to the limitations of existing technology, and as a result, virtual worlds need to be manually and laboriously constructed. Video2Sim is useful because virtual worlds can be used to train and evaluate AI systems. For example, videos of traffic accidents can be converted into simulations to test autonomous cars, or videos of kitchen scenes to test home robots. Simulation is more scalable and cost-effective than real world experiments and is particularly suited for machine learning algorithms that require a lot of training data. Furthermore, such an automated system can leverage a large number of videos to provide a comprehensive coverage of rare events, which is essential for evaluating and assuring the safety of autonomous systems. Therefore, Video2Sim has the potential to benefit a broad range of applications including robotics, healthcare, and transportation. Research in this project is integrated with K12, undergraduate, and graduate education through research training, course development and outreach events. This research develops key techniques toward a Video2Sim system with a focus on 3D shape and motion. This effort is organized into two thrusts: (1) reconstructing 3D shape and motion and (2) simulating dynamics and behavior. The goal of thrust 1 is to recover 3D shape and motion of a full scene from a monocular video, such that we can re-render the scene and re-enact the events from an arbitrary view. The focus is on developing methods to recover detailed 3D shape and 3D motion from arbitrary unconstrained videos. The goal of thrust 2 is to recover the underlying dynamics of a scene, such that we can not only re-enact the actual events but also simulate alternative outcomes. The focus is on developing methods to infer not only physical properties of passive objectives but also behavior models of agents, that is, entities that do not just move passively according to external forces but can plan and initiate their own actions. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.