Collaborative Research: Learning to learn with rigor: foundations of amortized inference

Purdue University, West Lafayette IN

Investigators

Abstract

This research project seeks to develop a rigorous theoretical foundation for amortized inference, a recent and impactful paradigmatic development in machine learning, statistics, and simulation. Amortization enables efficient, real-time responses to statistical queries by learning a model-dependent mapping from data to distributions, avoiding the need for expensive computations every time new data are presented. This capability underpins modern advances in generative AI, including diffusion models and variational autoencoders, with applications also extending to scientific machine learning (SciML) and simulation-based decision-making in operations research. Despite its widespread empirical success, fundamental questions persist: When do these methods work well, and when might they fail? How robust are the mappings to properties of the underlying problem? What kinds of statistical guarantees can be made about learned mappings, embodied for instance by deep neural networks? The goals of this project are twofold: (1) to deepen our understanding of the mathematical principles that underpin amortized inference, and (2) to inform the design of improved methods with provable guarantees. The project comprises three interrelated thrusts: 1) Functional Guarantees: This thrust investigates foundational properties of mappings from data to distributions: Do they exist? Are they unique? How well can they be approximated by, for example, neural networks? These results will elucidate the stability and robustness of amortized inference under data and model perturbations, 2) Statistical Guarantees: Building on the first thrust, this will establish both large-sample and finite-sample statistical guarantees for the learned mappings. The analysis will draw on techniques from M-estimation, approximation theory and Bayesian posterior contraction theory, 3) Methodological Developments: Existing amortized inference methods largely assume independent and identically distributed (i.i.d.) data. However, many applications-e.g., those involving data generated by Markov processes-violate this assumption. This thrust will extend amortized inference to non-i.i.d. settings, and will develop novel methodologies to fill this gap in the literature. Put together, these efforts aim to lay the theoretical groundwork for amortized inference, offering both insight and innovation in how statistical inference is carried out at scale. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.