CHS: Small: Data-Driven Material Understanding and Decomposition

Cornell University, Ithaca NY

Investigators

Abstract

We are in daily contact with a rich range of materials (metals, woods, fabrics, granites, etc.) that contribute to how we understand the world. Recognizing and modeling real-world materials have long been core challenges in computer vision and graphics. Recently, scene understanding has experienced an explosion of research activity driven by deep learning models trained on large-scale datasets. But the focus has mainly been on objects; materials have received less attention, and it has predominantly focused on careful measurements in laboratory settings. Of course, there is a large gap between materials in the real world and these laboratory settings. The PI's goal is to bridge that gap, to enable material understanding "in the wild." Toward this end, her group recently released large-scale crowdsourced datasets (OpenSurfaces, Intrinsic Images in the Wild, MINC) that are already being used extensively in the research community. Using this data to develop new material segmentations and recognition algorithms, the PI's team has produced state-of-the-art methods which open up new possibilities for data-driven material understanding that will impact a wide range of applications such as interior design, material editing, visual search, and robotics. Project outcomes (including new datasets, annotations, and code) will be made fully open and public. The PI actively mentors underrepresented minorities at Cornell, and is working with Women in Computing at Cornell (WICC) and Girls Who Code (GWC) to reach middle and high school students. This research will build prototypes of the annotation tools, and integrate them into summer workshops at Cornell aimed at high school minority students. The PI's group will also organize a Material Understanding Competition (MUC) to drive innovation in material recognition and segmentation, and intrinsic image decomposition. This project includes two major technical thrusts in material understanding: 1. Intrinsic images for material understanding. Intrinsic image decomposition aims to decompose images into intrinsic properties such as material and illumination. This decomposition is ill-posed and challenging for images in the wild. This work will collect new pairwise shading and depth annotations for intrinsic image decomposition; introduce a new perceptual metric to evaluate algorithms; solve for joint material recognition and intrinsic decomposition; and develop proof-of-concept applications for image-based editing using intrinsic image decomposition. 2. Material recognition for semantic understanding. Recognizing materials in the wild is extremely challenging. This work will collect large-scale material annotations with "click" data and train weakly supervised recognition algorithms; collect fine-grained material data for subcategories like wood and metal; develop new algorithms for coarse and fine-grain recognition; and develop proof-of-concept applications for intelligent material search, and material assignment to shapes.