CHS: Small: Learning to Automatically Design Interior Spaces

Brown University, Providence RI

Investigators

Abstract

People spend a large part of their lives indoors, in bedrooms, living rooms, offices, kitchens, etc. The demand for virtual versions of these spaces has never been higher; robotics, computer vision, architecture, interior design, virtual and augmented reality -- all of these fields need to create high-fidelity digital instances of real-world indoor scenes. To meet this need, this project will develop new generative models of indoor scenes that can rapidly synthesize novel environments. To achieve this goal, a scene synthesis system should be data driven, be able to quickly generate a variety of plausible and visually appealing results, and be user-controllable. While prior work has addressed indoor scene synthesis, no existing approach satisfies all of these requirements. Not only will this project achieve that goal, it also includes efforts to use the new software system for training robots to navigate. Broader impact of project outcomes will be enhanced through industrial technology transfer in collaboration with two furniture and interior design companies. The research will create freely available online demos, and will engage and mentor female students as research assistants. The first component of the envisaged system will be a new scene generative model based on deep convolutional neural networks that unifies a detailed, image-based representation of scenes based on floor plans with a discrete, symbolic representation of scenes based on object relationship graphs, thereby gaining the benefits of both to generate a variety of plausible scenes. Convolutions on both graphs and images will be employed to make synthesis decisions based on the relevant spatial context in the scene; the resulting model will be fast, controllable, and fully data driven. The system's second component will be a model of the visual compatibility of scene objects, which is necessary for generating visually appealing scenes. This model will exploit a convolutional network to analyze rendered views of the scene, capturing the visual appearance of the scene and objects in it; the network will be trained on a new dataset of professionally designed interior scenes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.