CIF: SMALL: Information Theoretic Foundations of Data Science

University Of California-San Diego, La Jolla CA

Investigators

Abstract

The advent of modern data generation, collection, communication, retention, and application has brought about an important new discipline termed data science. It draws upon diverse techniques from statistics, machine learning, computer science, and information theory to analyze, understand, and most importantly, utilize, data. It therefore significantly impacts diverse fields ranging from medicine to marketing, manufacturing, finance, and security. Much of the initial work in this area has centered on simple models to to facilitate analysis. This project is focused on bringing these results closer to real-world applications such as natural language processing. The results of this work are expected to lead to a better ability to predict future events, detect anomalies, and classify data. The research in this project is focused on three technical thrusts: (i) extending data science concepts to practical regimes, (ii) utilizing structure, and (iii) understanding the effect of memory in data sources. With regards to (i), the objective is to develop optimal distribution estimators not just for the traditional Kullback-Leibler divergence, but also for other metrics, and to do so for arbitrary parameter values, not just restricted asymptotic regimes. With regards to (ii), the objective is to utilize structure to improve distribution estimates and to resolve a conundrum where theorists and practitioners employ opposing distribution estimation techniques. With regards to (iii), the general objective is to study the problem of estimating distributions with memory. Preliminary results have uncovered an interesting dichotomy between compression and learning when memory is present. Through these technical thrusts, this project will study complex and realistic data models and derive results with important theoretical implications as well as applications in a variety of fields.