I-Corps: Using neural radiance fields (NeRF) and photogrammetry algorithms for creating 3D models

Massachusetts Institute Of Technology, Cambridge MA

Investigators

Abstract

The broader impact/commercial potential of this I-Corps project is the development of a three-dimensional (3D) scanning technology to visualize a space immersively using a smartphone. Currently, millions of people around the globe operate as content creators and viewers, sellers, and buyers. The proposed technology provides the ability to view large intricate objects and spaces. The expected first entry point for this application is e-commerce in real estate. The goal is to streamline the rental process for both landlords and tenants by leveraging artificial intelligence (AI)-driven matching, virtual AR tours, and digitizing the leasing experience. Additional applications are anticipated. This I-Corps project is based on the development of neural radiance fields (NeRF) and photogrammetry algorithms to create three-dimensional (3D) models with more detail and information than is available with images and videos alone. Currently, immersive technologies are cumbersome and require expertise and special equipment to build a remote experience. In addition, 3D reconstruction algorithms based on purely photogrammetry processing methods require large amounts of data (e.g., a room needs 1,000 images, while a medium scale historic building requires 100,000 images). In contrast, the proposed approach through NeRF enables 3D reconstruction with smaller data sets due to the ability of NeRF to use inference (e.g., a room needs 100 images and dramatically less time to 3D scan). From a user experience point of view, whereas images and videos present predetermined content, 3D models may allow users to simply explore by “walking through” complex spaces. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.