Adaptive Data Integration: Harnessing Commonality amidst Heterogeneity

Columbia University, New York NY

Investigators

Abstract

This project develops powerful new methods for integrating diverse and evolving datasets, a critical challenge in modern science and technology. In an increasingly data-driven world, information is often collected from many sources, at different times, and in various formats, making it difficult to analyze as a whole. This research will create flexible and reliable tools that can automatically integrate complex information. The tools will be applied to assess the safety of autonomous vehicles by combining limited test data from a new city with vast amounts of driving data from other regions. The project will also develop open-source software for all researchers and create educational materials to train the next generation of scientists and engineers. This research will establish dependable methodologies and solid theoretical foundations for data integration through three interconnected thrusts. First, the investigator will develop transfer learning methods for integrating samples from multiple sources to enable knowledge distillation and transfer across heterogeneous datasets. Second, the research will address the challenge of data streams that evolve over time by creating techniques that adapt to temporal distribution shifts. Third, the investigator will create principled approaches for learning latent structures by integrating multiple data views of the same subjects, such as combining social network information with individual user profiles. A key intellectual contribution of this work is the development of practical procedures that automatically adapt to unknown data heterogeneity with theoretical guarantees under minimal assumptions, overcoming a major limitation of current methods. The project will deliver innovative analytical tools, new theoretical insights into data integration, and open-source software packages to benefit the broader scientific community. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.