Computational and Mathematical Studies of Complexity Reduction Methods for Deep Neural Networks and Applications

University Of California-Irvine, Irvine CA

Investigators

Abstract

Deep neural networks (DNN) have become the state-of-the-art computing technology driving the recent advances in artificial intelligence, surpassing human performance on image and speech recognition tasks, and in playing complex games such as Go. However, deep networks typically consume billions of flops in computation and gigabytes of storage for model and data, rendering their deployment a challenge on mobile and energy limited platforms such as cellular phones and battery powered cars. The project aims to develop theory and algorithms for complexity reduction methods so as to maintain DNN's performance on low computational budget, achieving speed up and saving memory space. The project also studies light weight deep networks through automated architecture search and selection to reduce complexity at a higher design level. A broad range of applications include mobile computer vision, disease diagnosis and detection, face verification, as well as monitor and rescue missions by the drone. The project will actively involve graduate students and enrich their career development through both education and research activities. The approaches to be studied include (1) training of deep networks with low-precision weights and activation functions (so called quantization), (2) hand-crafted and automated lightweight deep networks, their training via variable splitting and their quantization. The training of quantized networks concerns with minimizing high dimensional discontinuous non-convex objectives under discrete constraints, for which novel coarse gradients and an accelerated technique (so called blending) will be analyzed to guide the descent and reach convergence. A differentiable treatment of discrete constraints, and of non-smooth and combinatorial structures will be fully developed. The methodologies and resulting algorithms from the project will contribute to information technology, optimization of civil infrastructure, smart and efficient mobile computing. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.