CAREER: Towards theoretical foundations of neural network based representation learning

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Building up good representations of input data has always been a central ingredient in machine learning. In computer vision, for example, one would like to have features representing the main objects in the image. In natural language processing, one would like to have features indicating the relationship between different words. A new paradigm shift in machine learning based on deep learning techniques has demonstrated the ability of machines to automatically learn good representations from the training data set without any prior knowledge. However, although these features are really useful for machines to learn the data set, are they actually "good" according to human standards? The project aims to contribute to the fundamental understanding of deep representation learning and inform the practical advancement of deep learning, improving its interpretability, robustness, and efficiency in large data regimes. The investigator will also develop a new graduate-level course and a public interactive software through the course of this project. The project aims to build a comprehensive theory for the new generation of neural-network-based representation-learning techniques. This includes the central questions of characterizing the statistical properties of the representations and how they are encoded in an actual neural network. This project has three major components. The first thrust is to characterize when would minimizing the training objective of the representation learning task leads to a unique representation in the neural network: leveraging the new theoretical development, the investigator will build up new training objectives that encourage such uniqueness. The second thrust is to theoretically study what representations can be efficiently learned by deep-learning models, and how are they encoded in the hidden weights of the neural networks after training. Finally, the investigator will study what statistical properties of the learned representations made them good for downstream tasks, which is critical to improving the interpretability of these neural network-based representations. Moreover, it will allow humans to better interact with deep-learning models for broader applications such as self-driving cars. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.