Information Theory with Directions: Geometric Structure and Coordinates on the Space of Probability Distributions

Massachusetts Institute Of Technology, Cambridge MA

Investigators

Abstract

In this research, we study a set of geometric approaches that can be used to fundamentally extend our methodology of information theoretical analysis. The conventional approaches to study information transmissions are based on the key quantities like entropy and mutual information, which can be viewed as a distance between distributions. Such approaches are particularly useful in studying static point-to-point communication problems, where relatively few probability distributions are involved. Facing the challenge of understanding large dynamic wireless networks, as many distributions with high dimensionalities are often involved, the conventional approaches often become cumbersome in solving or even describing the problems. The geometric approach studied in this work is in a sense a ?calculus? on the space of distributions. By developing notions of inner products, projections, and coordinate systems in the space of distributions, we add a sense of ?direction? in information theoretic analysis. The new insights from this approach often lead to better understanding and extensions to the existing network information theory results. The geometric approach is mainly used in two sets of problems. First, by visualizing the relation between multiple distributions, we develop the notion of divergence transition to describe the information exchange through general statistical coupling. This approach is particularly useful in studying mutli-terminal communication problems, in describing and controlling information contents at different terminals. Secondly, we used this approach to study error protections in dynamic communication channels. The conventional wisdom suggests that all information can be converted into bits and uniformly protected in transmissions, which is clearly inadequate for dynamic networked applications. In the study of new notions of heterogeneous information processing and unequal error protections, the geometric approach plays a crucial role.