NSF-BSF: CIF: Small: Self-adapting Code Generation in Rate-distortion Theory, Machine Learning, and Channel Coding

University Of California-Santa Barbara, Santa Barbara CA

Investigators

Abstract

This project builds on the investigators' early information-theoretic results, establishing a mechanism called 'natural type selection' for source coding, which adapts a randomly generated code and is shown to be asymptotically optimal. Fundamental expansions of this framework will be pursued to develop universally applicable methodologies, and thereby yield contributions to information theory itself alongside powerful learning techniques in other application fields, including wireless communications, content delivery, social media, artificial intelligence, and others. From the educational perspective, the project offers a training opportunity for graduate students to experience, first-hand, an international and interdisciplinary research collaboration, which combines theoretical depth with practical impact. It further offers opportunities for extensive curriculum enrichment, and to produce accomplished researchers and practitioners with capacities and skills that are in high demand. This project will develop novel approaches to learning, which employ universal self-adapting mechanisms for random code generation, designed to asymptotically achieve optimality for unknown source distributions. Research will be pursued, in terms of both theoretical analysis of performance bounds and powerful optimization approaches, along three main thrusts: i) Extension of the natural type selection framework to encompass continuous spaces and sources with memory, leveraging the concept of "parametric type" for continuous alphabets, which would expand applicability to virtually all practical scenarios of interest. ii) Applications in machine learning, where supervised learning (e.g., classification, regression) is reformulated as the rate-distortion problem of seeking the minimal amount of information to be learned from a source such that a desired output at the prescribed fidelity can be read from a random codebook; and, on the unsupervised learning side, where the "information bottleneck" method is reformulated universally in a self-adapting codebook generation setting. Both will leverage the optimization framework of deterministic annealing. iii) Applications in communications where stochastic mechanisms are developed for optimal channel input adaptation, including an important extension to multi-user communications which requires the development of a "distributed natural type selection" framework. This project is a collaborative effort between researchers in the US and Israel, with funding for Israeli researchers provided by the Bi-National Science Foundation (BSF). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.