SHF: Medium: Cross-Stack Algorithm-Hardware-Systems Optimization Towards Ubiquitous On-Device 3D Intelligence

Georgia Tech Research Corporation, Atlanta GA

Investigators

Abstract

Deep neural network (DNN)-boosted three dimensional (3D) reconstruction promises to become the next tech disruptor and revolutionize many aspects of human life and means of production. It is expected to reach a value of $1575.64 billion by 2028. This is because 3D-reconstructed data is rich in scale and geometric information, thus enabling 3D intelligent systems to achieve much better machine-environment understanding and perform newly possible functionalities. In parallel with the growing need for 3D reconstruction, the number of daily-life devices has been booming and is estimated to reach 500 billion by 2030. Therefore, there has been a monumental demand for bringing 3D reconstruction-enabled intelligent functionalities into numerous (heterogeneous) devices, ranging from drones to self-driving cars, augmented reality (AR) and virtual reality (VR) devices, and many more, for enabling ubiquitous 3D intelligence. Despite the big promise of ubiquitous on-device 3D intelligence, a vast and increasing gap still exists between the prohibitive complexity of powerful 3D reconstruction algorithms and the constrained resources in commonly used devices. This is because the increased data-, model-, and training-level complexity required in 3D reconstruction-enabled intelligent functionalities/models exponentially aggravates their computational cost as compared to 2D intelligent ones. Furthermore, 3D reconstruction models feature very unique computational and data access patterns compared to those of 2D image-based DNN models. This project aims to close the aforementioned gap and foster a systematic breakthrough for enabling on-device 3D reconstruction-enabled intelligent functionalities through a holistic exploration of a joint optimization and harmonization of algorithm-, hardware-, and system-level innovations. The results of this project will culminate in the creation of innovative course materials and open educational resources designed to engage a diverse student body, fostering an inclusive learning environment and serving as a springboard for creativity and innovation. The project will advance knowledge and produce scientific principles and tools for enabling on-device 3D intelligence. First, Thrusts 1 and 2 will develop the fundamental underpinnings of dedicated algorithm-hardware co-design solutions for enabling instant training and real-time inference of on-device 3D reconstruction. In particular, Thrust 1 focuses on per-scene optimization scenarios and Thrust 2 concentrates on cross-scene scenarios considering that a sufficient number of captured views of the target scene is not always available. Second, Thrusts 3 and 4 will develop algorithm-system co-optimization solutions for enabling resource-adaptive on-device 3D intelligence. Specifically, Thrust 3 deals with privacy-sensitive/proprietary on-device training data by extending to distributed learning of cross-scene 3D reconstruction models over heterogeneous edge devices, while Thrust 4 explores resource adaptive inference scheduling methods by leveraging algorithm-system co-design. Finally, an integration effort is conducted to evaluate the innovations of the four thrusts and demonstrate their benefits in realistic systems. Overall, this project is geared towards significantly advancing the state-of-the-art of on-device 3D reconstruction by improving model accuracy, efficiency and enabling real-time inference and learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.