MFAI: Mathematical Foundations of Alignment in Generative Artificial Intelligence

University Of Pennsylvania, Philadelphia PA

Investigators

Abstract

Generative large language models (LLMs) and generative diffusion models (GDMs) have become known for generating data that can have an astounding resemblance to human-generated content. Yet, the content generated by these models can introduce serious risks in specific applications. These models are known to replicate biases of their training data, produce unsafe outputs, and generate content that is misleading, false, and reprehensible. This project tackles these challenges within the general framework of alignment. Large pretrained models for image and language generation are available in the public domain but they are generic. It is of interest to most users to retrain these models to adapt them to their specific goals and principles. Our success will make it possible to better incorporate, among others, fairness, safety, reliability, robustness, and truthfulness requirements. This is a necessary development for tools that will be deeply integrated into the social and economic fabrics of our country. Our technical approach builds on three properties of alignment problems in generative AI: (P1) Alignment problems are reinforcement (RL) problems in which the value function is known. This makes alignment easier than generic RL because most of the typical complications of general RL problems are related to the learning of the value function. (P2) Alignment problems in generative language models and generative diffusion processes share the same structure. The objective is to align a generative model to user requirements given a prior reference in the form of a pretrained generative model. (P3) Alignment problems are highly nonconvex in the parameter space of deep neutral networks or transformers. However, they are strongly concave in distribution space. Property (P1) shows that alignment in generative AI is simpler than often thought. Property (P2) motivates an integrated research program on constrained generative AI that encompasses its two extant versions. Property (P3) is the fundamental key to our proposed research. Optimization and statistical foundations of constrained generative AI will be developed by drawing connections between the (strongly convex) problem in distribution space and the (highly nonconvex) problem in parameter space. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.