Quantization and Compression for Neural Networks: Theory and Algorithms

University Of California-San Diego, La Jolla CA

Investigators

Abstract

The research project focuses on making advanced AI technologies, like the neural networks used in language models and for image generation or analysis, more efficient and environmentally friendly. Today, these technologies require significant computational power, which makes them both costly and energy-intensive. This project aims at developing algorithms for replacing such networks with functionally equivalent ones that require fewer computational resources, without losing their effectiveness. The resulting reduction in computational complexity will also help enable the use of AI in real-time applications and on devices with limited resources, further expanding this critical technology's reach. Thus, the expected scientific outcome is the development of robust, efficient, practical algorithms that are also backed by rigorous theoretical guarantees. Moreover, we anticipate that the theoretical tools we develop in order to analyze these algorithms will find broader use in other application areas. The project also emphasizes the importance of education and academic community involvement. By integrating research findings into university courses and involving students from potentially diverse backgrounds, the project will help prepare the next generation of mathematicians and engineers. Additionally, it aims to share breakthroughs with broader communities through journal publications, workshops, and conferences, while also connecting students with real-world industry applications. As previously stated, the project aims to address the challenge of compressing large neural networks, which are pivotal in modern AI applications but are resource-intensive. Thus, the research will focus on developing algorithms that reduce the computational demands of these networks by minimizing the memory and power needed without compromising their performance. Our approach involves three main strategies, quantization, pruning, and low-rank approximation. Among these, quantization transforms neural network parameters into formats that require fewer bits, thus reducing memory usage and computational intensity. Meanwhile, pruning selectively removes less important parameters from the network to streamline computations. Finally, low-rank approximation replaces large matrices representing the weights in the network with products of smaller matrices, in a way that retains essential information while requiring less memory and computation. We will develop algorithmic approaches for quantization, pruning, and low-rank approximation that are underpinned by rigorous mathematical theories to ensure the reliability and effectiveness of the compressed models. In our analysis, we will utilize stochastic process theory, geometric functional analysis, discrete geometry, discrepancy theory, optimization theory, compressed sensing, and dimensionality reduction, to name a few. These diverse areas will help us establish a solid theoretical foundation for our algorithms consisting of lower bounds on best-possible theoretical error guarantees as well as upper bounds on the errors resulting from our algorithms. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.