HCC: Small: 3DStylus: User-Guided Shape Manipulation using Neural Priors

University Of Chicago, Chicago IL

Investigators

Abstract

Digital 3D models are used in a wide variety of applications, including manufacturing, engineering, medicine, and entertainment. The rapid growth in demand for 3D models, however, has outpaced our ability to construct them. Employing current modeling tools to perform even the most basic of 3D operations, such as selecting a region on a shape, requires years of extensive training and experience. This project will develop new and accessible tools and techniques for creating and manipulating 3D objects, eliminating existing technical barriers of entry and thereby democratizing 3D content creation. Project outcomes will have broad impact by making 3D modeling more accessible, empowering both expert and lay users to create and transform objects for applications across a wide range of industries and professions. The project is centered around the development of 3DStylus, a suite of foundational tools that will enable users to edit and modify existing 3D shapes and create new ones from scratch, using simple text as input. 3D Stylus will encompass three central components: (i) 3D Editor, which will enhance existing 3D models by incorporating text-specified textures, materials, and localized modifications that preserve the underlying shape; (ii) 3D Morpher, which will transform existing 3D models into new text-specified geometry, while preserving the original texture details; and (iii) 3D Creator, which will allow users to intuitively create novel 3D geometries, and make desired edits and manipulations, using only a text description. The research will leverage deep learning techniques which have been used so successfully in 2D (as evidenced by the revolutionary impact of DALL-E), but which have generally been out-of-reach in 3D given the relatively small amount of available high-quality 3D data. The novel approaches envisioned will instead use the abundantly available 2D datasets as a signal for editing 3D objects. Thus, the transformational potential of deep learning can be harnessed to achieve advanced 3D modeling capabilities, without the need for large 3D datasets. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.