Making Use of the Curse of Dimensionality in Modern Data Analysis

University Of California-Davis, Davis CA

Investigators

Abstract

This research project delves into cutting-edge data analysis, frequently dealing with high-dimensional or non-Euclidean data, such as sensor readings, genomic information, imagery, and network datasets. Such data is commonly encountered across diverse disciplines, including biology, social science, computer science, and astronomy. A major challenge in analyzing this data is the curse of dimensionality, which causes traditional tools to degrade rapidly as the number of dimensions or features grows. While previous attempts have focused on reducing the dimensionality of the data or through regularization techniques, these methods often exhibit limitations. In contrast, this project adopts an innovative strategy by harnessing the patterns that emerge as a result of the curse of dimensionality to bolster data analysis. The project aims to provide valuable tools for data analysis and explore the role of statistics in the era of big data. The tools developed within this project will be made available as open-source software packages with thorough documentation, enhancing collaboration between the statistics community and researchers from various scientific fields and making data analysis procedures more transparent. The project also includes training and educational components for undergraduate and graduate students, equipping them with interdisciplinary data analysis skills that will be invaluable for the next generation of researchers. This project seeks to develop innovative methodologies and foundational theories for crucial data analysis tasks involving high-dimensional and non-Euclidean data. Specifically, the project will create a pioneering high-dimensional classification framework that leverages interpoint distance ranks and takes into account the curse of dimensionality, resulting in significantly reduced misclassification rates compared to existing methods across various settings. Moreover, the project will establish a unified community detection framework capable of identifying all three community structures -- assortative, disassortative, and core-periphery – without prior knowledge of which community structure the network is. By linking these distinct structures to high-dimensional data behaviors, where the core-periphery structure naturally emerges due to the curse of dimensionality, the unified framework exhibits superior performance in numerical studies on simulated and real datasets across all three mixing patterns, whereas existing methods struggle in at least one of these patterns. Lastly, the project will develop an innovative high-dimensional clustering framework that employs the patterns of curse of dimensionality to reduce misclustering rates. These methodological and theoretical advancements will enhance the understanding of modern, complex data from diverse fields, promoting the comprehension of significant scientific issues in these areas. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.