Collaborative Research: Geometric Analysis and Computation for Generative Models

Duke University, Durham NC

Investigators

Abstract

Research in unsupervised learning and generative models is concerned with uncovering structure and relationships in data with the intent of being able to generate new, as yet unseen, examples of the data set. Generative models learn the distribution of a data set from finite samples and provide an efficient sampler of the approximated density, rather than relying on labels for supervision. These models are a powerful tool for analyzing large volume, high-dimensional data in an unsupervised way. While generative models are an active research topic in machine learning, many theoretical and computational questions for such models remain unclear. This collaborative research project will study generative models from a geometric perspective, focusing on both performance guarantees and efficient implementations. The ability to efficiently create new data points that are guaranteed to be similar to the existing data has important implications in a variety of applications, including medical data analysis and privacy, bioinformatics, modeling of image and audio signals, and general high-dimensional data analysis in which it is difficult to collect labeled data for supervised algorithms. The ideas and approaches in this research project center around the techniques that have evolved in the manifold learning field over the past decade. These mathematical tools, in particular local neighborhood preserving maps, approximation analysis in terms of intrinsic dimensionality, and construction of global coordinate systems based upon local affinity, have natural applications in the study of generative models. The project is comprised of four fundamental questions that arise in the field: (a) What are the types of distributions that generative networks are capable of learning efficiently, and how does the intrinsic dimensionality of the distribution affect convergence? (b) How can non-parametric generative models be created for dimension-reduced representations that arise in manifold learning, and which only depend on the intrinsic geometry of the data? (c) How can efficiently-computed metrics be defined between high-dimensional distributions for use in assessing the validity of various generative models? (d) How can these metrics be used to examine the various paths generative models take through the parameter space while being trained, and what clusters of starting points give optimal generators? The project will focus on both mathematical and computational aspects of these problems, aiming at resolving fundamental questions about these tools that are widely used in various data analysis and signal processing applications in science and industry. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.