Collaborative Research: CDS&E-MSS: Deep Network Compression and Continual Learning: Theory and Application

Northeastern University, Boston MA

Investigators

Abstract

Deep neural networks (DNNs) have led to transformative developments in a wide number of tasks, such as identifying people or objects in photographs, translating and synthesizing natural language, and generating scientific and medical images. These advances were possible because neural networks are trained to find patterns in large datasets. Balancing the trade-off between the size and performance of a deep network is a vital aspect of designing deep neural networks that can easily be translated to hardware. Although deep learning yields remarkable performance in real-world problems, they consume a large amount of memory and computational resources, which limits their large-scale deployment. Once neural networks are trained, they can also be brittle in the sense that when a network is trained on a new task, it typically forgets previously learned tasks. This brittleness presents challenges in building artificial intelligence systems that are intended to learn continually throughout their lifespan, and it leads to even more computational resources spent to retrain networks on tasks they have already learned. This results in large demands for electrical power, which leads to an increased carbon footprint and adverse environmental impacts. In this project the investigators propose novel algorithms and theoretical analysis for reducing the power consumption of neural networks by compressing their learned parameters and using these compressed parameters for continual learning. Graduate students will be involved in the research and receive interdisciplinary training. The overall goal of this project is to develop a novel probabilistic framework for neural network compression. Using this framework, the investigators will develop network compression algorithms based on the connectivity between filters and layers, which provides a sparsification criterion that is efficient in both training and testing processes. After network compression identifies which parameters of a neural network are more important than others, this feature can be used to develop algorithms for continual learning which are efficient because the important parameters are prioritized for the learning of subsequent tasks. The investigators will develop compression-inspired algorithms for continual learning based on statistics of individual layers and the connectivity between layers. The investigators will also provide a sparsity analysis and theoretical explanations in the form of mathematical theorems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.