Energetic Variational Inference: Foundations, Algorithms, and Applications

Illinois Institute Of Technology, Chicago IL

Investigators

Abstract

Variational Inference is a powerful tool used to boost efficiency and flexibility in machine learning and artificial intelligence algorithms, particularly those based on large amounts of data. In this project, the investigators plan to create a unified and systematic framework for variational inference methods, making two key contributions. First, the investigators will establish the theoretical foundations for the proposed framework, which will support and justify using existing and new variational inference algorithms in machine learning applications. Second, the investigators will provide a systemic procedure to create new variational inference algorithms and apply them to emerging machine learning problems. In addition to these new scientific developments, the investigators will create new courses and workshops on machine learning, recruit both undergraduate and graduate students for summer, project-based research programs, and provide mentorship to local high school students through hands-on machine learning training programs. Collaborations are planned with industrial data science partners to apply these new algorithms in practice and to train the workforce with the start-of-the-art machine learning tools. The proposed "Energetic Variational Inference" framework is based on an energetic variational approach, which has been successfully used to study complicated non-equilibrium systems in physics and biology. The investigators will provide a blueprint for generating new algorithms by introducing various options for the four essential components of the proposed framework: the divergence functional, the dissipation functional, the representation of the probability density, and the temporal discretization. The investigators will study convergence in the continuous formulation as well as estimate the error bounds after temporal discretization of the underlying continuous dynamic system. More importantly, these theoretical results can be applied or extended to other flow-based variational inference approaches. These methods will be applied to problems in supervised learning, density estimation, and generative learning. Additional novel applications in machine learning, statistics, and statistical physics will also be developed. The algorithms will be packaged into open-source software for public use. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.