CRII: RI: Reasoning Geometric Commonsense for 3D Image/Video Parsing

San Diego State University Foundation, San Diego CA

Investigators

Abstract

Commonsense reasoning studies the consensus reality, knowledge, causality, and rationales available to the overwhelming majority of people, and can be used to enhance all aspects of Artificial Intelligence (AI). This project develops representations of geometric commonsense as well as computing principles of commonsense reasoning for computer vision applications. The project systemically studies commonsense knowledge over geometric dimensions of scene entities, e.g., the length of a sedan is shorter than that of a bus; or that window edges on the same façade are parallel to each other and are orthogonal to the edges on the ground. These first-order and second-order knowledges, once extracted, are fairly stable across different types of scenes, and are informative enough for enhancing the understanding of images or videos in both 2D and 3D. The project integrates research with education by supporting graduate students, and outreaches to computer vision and AI research communities by organizing workshops in the relevant conferences. This research studies geometric commonsense reasoning for 3D scene parsing in images or videos, and contributes a unified probabilistic approach that is capable of reconstructing a wide variety of scene categories (e.g., suburb, urban, campus) from a single input image or a monocular video sequence. The project approaches the problem from two aspects. First, a new attributed grammar model is developed to represent both images and the associated geometric commonsense knowledge using a hierarchical graphical structure. With this grammar model, the segmentation of semantic regions, the reconstruction of scene entities, and the reasoning of geometric commonsense can be all solved through creating a valid parse graph from images or videos. Second, a new computing framework is introduced so that the inference of image parsing can be conducted in the joint space of discrete semantic labels and continuous geometric labels, and the learning of grammar models can be conducted over training images with weak supervision. The developed techniques enable a state-of-the-art computer vision system that can robustly estimate semantic and geometric scene structures from images or videos.