Information Processing Theory and Applications

William Marsh Rice University, Houston TX

Investigators

Abstract

PROPOSAL #0105558 WILLIAM MARSH RICE UNIVERSITY PI: JOHNSON, DON H. The closely allied fields of signal processing and information theory have never found common ground. Signal processing focuses on how signals can represent information and how systems manipulate and change signal structure. Information theory revolves around the structure of information that signals represent, but ignores what information is meaningful to the receiver by concentrating on efficient compression and communication. The research develops a new theory of information processing that weds these two disciplines and has the dual goals of understanding how effectively signals, no matter what their nature, can represent information and of quantifying how well systems process information. Because of the theory's generality, we analyze both communication systems, to probe how effectively they convey information and meaning, and neural processing systems, to understand how neural groups process and represent information. We quantify how well signals represent information by computing an information-theoretic distance (it obeys the Data Processing Theorem) between signals associated with two instances of the encoded information. We assume that the signals are stochastic, and the distance measures how different are the probability distributions associated with the signals. We use the Kullback-Leibler distance because it is related both to optimal classifier performance via Stein's Lemma and to optimal least-squares estimator performance through the Cramer-Rao bound. A larger distance thus corresponds to a more effective representation of the information. The information processing ability of a system is measured by the information transfer ratio, defined to be the ratio of distances computed at the system's input and output. With this ratio, we quantify how well an information processing system behaves as an information filter.