Computational and Mathematical Studies of Compression and Distillation Methods for Deep Neural Networks and Applications

University Of California-Irvine, Irvine CA

Investigators

Abstract

This project will develop efficient deep learning architectures for the deployment of artificial intelligence algorithms on resource limited platforms, such as mobile computing and the internet of things. High performance deep neural networks consume hundreds of billions of flops in computation and store hundreds of millions of parameters in memory. However, devices with limited resources with respect to both power and memory call for constructions of lightweight deep neural networks to maintain the performance level of their heavyweight counterparts. This project aims to develop an efficient search-based architecture compression method and a novel teacher-tutor-student (knowledge distillation) framework to extract a smart lightweight network (student) from a state-of-the-art heavyweight network (teacher) with the help of an intermediate network (tutor). Real world applications benefitting from the project include visual computing on mobile phone and autonomous driving, the delivery, monitor and rescue missions by the drone, and disease detection and diagnosis in mobile health. The project will train graduate students and enrich data science curriculum for a diverse body of undergraduate students in science and engineering at minority serving institutions. The project will study a dual-network cooperation method for the search-based architecture compression so that the low level network weights and high level network structures are both optimized efficiently. A key element is a relaxation of bilevel optimization to a single level optimization task together with non-differentiable decision-making in the search approximated by a differentiable proxy function. The project will advance knowledge distillation methods, expanding distillation to intermediate layers of teacher networks by leveraging similarity measures on tensors of different shapes, and multi-resolution path learning techniques arising in image segmentations. The investigator will also formulate simplified classification problems for mathematical analysis and the understanding of distillation learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.