CIF: RI: Small: Information-theoretic measures of dependencies and novel sample-based estimators

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

Measures of dependencies play central roles in discovering associations between variables that leads to scientific discoveries. In practice, analysts need to compute these measures from data, which can be challenging. The standard estimators can fail when, for example, the data has a mixture of continuous and discrete variables, or when the data lies on a complex space with abundant boundaries. The aim of this project is to address practical issues in estimating measures of dependencies, and provide novel estimators to overcome these challenges. The success of the proposed work will result in novel estimators for discovering new aspects of data. The immediate impact is in two specific contexts: discovering correlations in biological datasets and analyzing the inner-workings of deep neural networks; the lasting impact will be in diverse fields including genomic, biology, machine learning, and artificial intelligence. This project also integrates research with education through the creation of a graduate course on statistical learning. In addition, the project will offer undergraduates the opportunity to be involved in research. This proposal addresses two fundamental questions: designing novel estimators for information theoretic measures and designing novel estimators for modern measures of correlation that is defined as a solution of optimization problems. In the former, two major challenges are addressed: variables of mixed type (continuous and discrete) and boundary biases. Borrowing techniques from local log-likelihood density estimators, nearest neighbor methods, and order statistics, this leads to a new estimator that can adapt to the local geometry of the distributions in a principled way, that improves significantly over existing estimators. In modern data analysis, several measures of correlations are naturally defined as solutions of optimization problems, making them challenging to estimate. This proposal aims to provide a principled approach and propose a new estimator borrowing insights from importance sampling and nearest neighbor methods. The proposed framework is applied to estimate hypercontractivity ratio, an information theoretic quantity that captures hidden correlations in the data and is naturally defined as a solution of an infinite dimensional optimization. The proposed measure of hypercontractivity is shown to discover potential correlations that other standard measures are not able to, in canonical synthetic examples and real datasets. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.