Collaborative Research: CIF: Medium:Data Manifolds and Local Geometry in Deep Networks

University Of Southern California, Los Angeles CA

Investigators

Abstract

Over the last decade, machine learning (ML) and artificial intelligence (AI) have often been in the news, thanks to striking achievements in areas such as self-driving cars and AI-enabled chat services. Much of the progress that has led to these breakthroughs can be attributed to the development of deep learning methods (deep neural networks). While these methods have developed very quickly to address a large number of tasks, training them often involves manual tuning and requires enormous amounts of data, resulting in significant costs, including both manpower and power consumption. More sustainable, lower-cost approaches for developing and deploying complex ML and AI systems are needed to ensure that these methods can be widely deployed to accelerate scientific progress and benefit society. For this purpose, it is important to develop a better understanding of why and how certain neural networks outperform others. This is critical for faster prototyping, reduced training times, lower complexity, and better interpretability of deep learning models. The investigators will test their methods through external collaborations and they plan to develop multiple activities to broaden participation in computing, including new course development, REU programs, and K-12 activities. This project focuses on developing techniques for a mathematical understanding of deep learning intrinsic to the given task and data being considered. In this project, neural networks are viewed as operators whose behavior is characterized in terms of their action on the training (or testing) data and, more specifically, on the geometry of the data. One major benefit of this approach is that it can be applied to a wide variety of networks since it abstracts the architecture and considers only how a specific system converts inputs into outputs. This project has three main research objectives. First, when both the model and data are fixed, the goal is to compare different networks and understand how they learn and generalize to unseen data. Second, this project investigates the situation where the data is fixed, but the network model can change. This work focuses on clustering, dimensionality reduction, and importance sampling to reduce the size of the model. Finally, the geometric analysis developed in this project is applied in settings where both the model and data change. This makes analyzing domain adaptation and graph neural network transfer possible, providing efficient algorithms and generalization error guarantees. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.