CAREER: Entropy Geometry in Variational Inference Signal Processing

Drexel University, Philadelphia PA

Investigators

Abstract

The past decade has proven that variational approaches to approximate Bayesian inference hold the key to pragmatic solutions to many signal processing problems previously thought to be fundamentally difficult. Important signal processing problems where the application of appropriate variational approximate Bayesian inference techniques led to significant breakthroughs include the decoding of capacity approaching codes for noisy channels, underdetermined speech source separation, and distributed estimation over networks. These applications provide evidence that variational inference techniques have revolutionized the state of the art in signal processing because of their ability to provide high performance estimates at reasonable complexity and communication (equivalently, energy consumption) costs. However, the tradeoff between their performance and required complexity and communication, the very phenomenon leading to their widespread and growing adoption, remains incompletely understood. The underlying thesis of this project is that entropy & information geometry lies at the heart of the tradeoff between performance, complexity, and communication in variational inference based signal processing. The research being performed improves our understanding of entropy geometry, which primarily dictates the fundamental relationship between the performance of a collaborative estimation algorithm and its communication cost. This research also develops the information geometric relationship between a variational inference signal processing technique's performance and complexity. Together, these insights allow algorithms for robust speech processing and collaborative estimation over networks to be developed that provide optimal performance at a tunable complexity and communication cost. In addition to its benefits in each of these areas, the work has a synergistic aspect in that it contributes directly to the creation of an overall design science for efficient distributed variational inference signal processing algorithms.