Representation Learning via Variational Mean Field Theory

Purdue University, West Lafayette IN

Investigators

Abstract

Deep learning has had tremendous success in many science and engineering applications. It revolutionizes classical model-based methods using data-driven approaches in a computationally tractable way. Representation learning, which extracts useful information from raw data needed for downstream tasks (for example, classification, regression, and reinforcement learning), is one of the most important directions of deep learning. Despite its success in areas such as image and signal processing, speech and object recognition, natural language processing, chemistry and drug discovery, theoretical understanding of representation learning is far from satisfactory. This project aims at new understanding of representation learning (in particular, deep generative models and graph representation learning) using mean-field game (MFG) theory. The research is intended not only to provide a theoretical understanding of generative models and graph representations but also to serve as a key step in transforming deep representation learning from a black-box approach to an explainable and trustworthy method. The project will benefit researchers in academia, government labs, and industry and will provide interdisciplinary training in applied mathematics, engineering, and data science to undergraduate and graduate students. Collaboration with the MIT-IBM Watson AI Lab, which offers complementary technical skills and industrial angles, will enhance career opportunities for undergraduate and graduate students. This project aims at i) bridging theoretical and analytical tools in MFG with deep generative models and graph representation learning, and ii) exploring new data-driven architecture designs in MFG-guided representation learning through the lens of bi-level optimization. This first objective is to understand the practical normalizing flows as the variational MFG, where reversible particle trajectories in MFG can be naturally viewed as the generative and normalizing directions in normalizing flows. The second objective is to propose a new framework for graph representation learning based on MFG. This new way of modeling with graph-structured data overcomes the limitation of the message passing framework studied from the graph isomorphism angle, leading to new network architectures that are faster and more scalable to train. To complement the expert-based choice of the dependencies and architectures in MFG, a new bi-level optimization approach will be investigated to jointly learn model dependences, architectures, and parameters in the first two objectives. The complementary expertise of the research team is being leveraged to enrich the theoretical foundations of deep learning through MFG-, graph-, and optimization-based approaches. This research agenda is expected to foster multidisciplinary efforts at the intersection of representation learning, graph learning, bi-level optimization, signal processing, and control theory. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.