CHS: Small: Deriving and Exploiting Shape Semantics

Stanford University, Stanford CA

Investigators

Abstract

3D representations are the most faithful digital encoding of physical objects, allowing us to store and manipulate all kinds of information about the object, both high-level (e.g., affordances and functionality) and low-level (e.g., appearance and material). Furthermore, they do so in a way that is more complete than 2D images or entirely symbolic representations such as text-based knowledge graphs. Yet associating semantic information with 3D models is not easy, because data that directly links 3D models to their function and use, their semantic parts and attributes, is not widely available. Given the emergence of 3D shape repositories on the Web supporting a variety of applications (such as 3D printing), plus the availability of affordable 3D scanners and their incorporation in computers and mobile devices, the time is ripe to enrich these repositories with semantic information that will vastly increase their accessibility and usefulness beyond the specialized applications for which they were originally developed. The PI's goal in this research is to do just that, by developing mathematical and algorithmic techniques which extract, encode, and exploit the semantics of 3D models; these in turn will be combined in a novel search engine for 3D models that will exploit semantic shape attributes in a unified way and will allow much broader access to and use of 3D repositories, thereby ultimately supporting many commercial applications while also proving useful to other research communities. Toward these ends, the PI will acquire semantic information about shapes by a combination of geometric analysis and user annotation. The geometric analysis is not only of shapes in isolation, but instead a joint analysis of collections of related shapes. The aim is to build networks among 3D models that can transfer information between them. Through these networks, using novel mathematical techniques, the PI will enable understanding of the shared structure as well as the variability of shapes in a collection. Common parts or structures in shapes invariably have semantic significance, and the role of a shape in a collection (its relationships to its partner and peer shapes) often defines semantic attributes of the shape. Since user annotations are expensive to obtain, the plan is to develop tools that exploit the above-mentioned networks so that only a modest number of annotations need to be obtained by crowd-sourcing queries. However, user annotations will be sparse and noisy, so the PI will also develop techniques for cleaning them up, as well as for propagating and aggregating them. Understanding how to integrate semantic structure reflected in the geometry of shapes with semantic structure reflected in language is one of the deep problems the PI will tackle in this research.