Tackling High Dimensionality for Modern Machine Learning: Theory and Visualization

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

This research project aims to address the recent challenges of modern machine learning from a statistical perspective. Deep Learning and particularly Large Language Models have the potential to transform our society, yet their scientific underpinning is much less developed. In particular, large-scale black-box models are deployed in applications with little understanding about when they may or may not work as expected. The research is expected to advance the understanding of modern machine learning. It will also provide accessible tools to improve the interpretations and safety of models. This award will involve and support graduate students. The project is motivated by recent statistical phenomena such as double descent and benign overfitting that involve training a model with many parameters. Motivated by the empirical discoveries in Deep Learning, the project will develop insights into overfitting in imbalanced classification in high dimensions and the effects of reparametrization in contrastive learning. Understanding the generalization errors under overparametrization in practical scenarios, such as imbalanced classification, will likely lead to better practice of reducing overfitting. This project will also explore interpretations for black-box models and complicated methods: (1) in Transformers, high-dimensional embedding vectors are decomposed into interpretable components; (2) in t-SNE, embedding points are assessed by metrics related to map discontinuity. By using classical ideas from factor analysis and leave-one-out, this project will result in new visualization tools for interpretations and diagnosis. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.