RI:Small:3D Shape Understanding and Generation using Unstructured Point Clouds

University Of Massachusetts Amherst, Amherst MA

Investigators

Abstract

3D shape understanding and generation are fundamental research problems in computer vision and graphics. Techniques to efficiently and accurately detect, classify, segment, and label 3D shapes have widespread applications in autonomous vehicles, home assistance, industrial automation, and medical engineering. Similarly, techniques to generate new shapes or complete partial shapes are instrumental in design, manufacturing, and digital preservation. While recent advances in deep learning have revolutionized 2D image understanding and generation, adapting them to 3D shapes is still a fundamental challenge. One bottleneck is the choice of 3D representation. Standard choices such as 3D voxel grids, geometric images, multi-view representations have been extensively studied, but they impose various limitations such as poor efficiency, issues with representing complex shapes, difficulties in capturing interior structures and maintaining view consistency. This project investigates suitable and efficient 3D representations that enable deep learning algorithms to achieve the level of success on 3D shapes as on 2D image understanding and generation tasks. This project aims to study shape understanding and generation using unstructured point clouds, which represent 3D shapes as a collection of points on shape surfaces. Such representation is compact, scalable, and suitable for capturing complex geometric details. For shape understanding, the research investigates new approaches, such as using various spatial data structures and multigrid tree networks, to enable deep convolutional networks on unstructured point clouds. For shape generation, the research studies effective generative models, such as using variational autoencoders and GANs, applied on point clouds to produce high-quality 3D shapes in a scalable manner. Different choices of loss functions on point clouds will be investigated in detail. The research will be evaluated on real-world datasets for shape recognition tasks, image-to-shape reconstruction and shape completion tasks. It also enables new applications such as unsupervised 3D shape induction from unlabeled image collections in the wild. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.