CAREER: Learning Structured Representations with Deep Probabilistic Programs

Northeastern University, Boston MA

Investigators

Abstract

Programming languages can play a decisive role in democratizing machine learning research. In deep learning, programming frameworks have made it possible – and even routine – to define neural networks in a modular manner. This has led to an explosion of research, with breakthroughs in computer vision, natural language processing, and reinforcement learning. The proposed work will develop deep probabilistic programming languages, which train neural networks to perform inference in simulation-based models. These languages will help the community address emerging challenges in artificial intelligence research by developing models that incorporate inductive biases to reason about uncertainty and improve generalization from limited data. In applications in the physical sciences, inductive biases can incorporate our physical knowledge of a problem domain. More generally, probabilistic programs help us represent model structure, for example to reason about how actions affect objects in a scene. The technical challenge that the proposed work addresses is scaling up methods for inference in probabilistic programs. To do so, the investigators will develop a language for inference programming, which will allow users to optimize the inference approach for a specific model. Inference methods reason about the posterior distribution over unknown variables in a program in light of observed data. Stochastic variational methods approximate the posterior by training a neural network that accepts data as input and returns a distribution over variables. This strategy works well in simple models in which unknown variables take the form of an unstructured vector. However, in models with more complex structure, efficient inference often requires reasoning about conditional independence. This is challenging for programmatically specified models, where reasoning about model structure requires program analysis. To address this challenge, the investigators will develop an inference language based on two constructs. The first are model combinators, which define a first-order language for composing black-box programs in a manner that allows us to reason about conditional independence. The second are inference combinators, which may be used to apply correct-by-construction importance sampling operations to specific components of the model. Together, model and inference combinators will allow users to develop correct and efficient stochastic variational methods for specific models. In addition to developing these fundamental abstractions and proving their correctness, the investigators will demonstrate the utility of these methods in applications to few-shot deep generative models, and structured energy-based models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.