Scalable Entity Resolution for Massive and Streaming Data Contexts

Colorado State University, Fort Collins CO

Investigators

Abstract

This research project will develop the first model-based statistical approach to perform entity resolution for streaming data contexts and will address the issue of scalability in both online and offline scenarios. With the ubiquity of data, linking multiple data sets is a crucial first step in many types of inference for myriad applications including healthcare, official statistics, ecology, and fraud detection and national security. Entity resolution is the task of resolving duplicates in two or more partially overlapping sets of records, or files, from noisy data sources without a unique identifier. Statistical approaches to entity resolution are advantageous because they provide interpretable parameters and quantify uncertainty in the linked records. Linking becomes more challenging when the data update over time, termed streaming or online data, or when the data scale is massive. Currently no statistically model-based approaches exist to resolve entities in an online way. Relatedly, the nature of streaming data exacerbates the challenge of scalability in that the number of records to be linked accumulates. The methods will be made accessible to practitioners and other researchers through open-source software and the project will additionally provide educational and professional training and mentoring to graduate students. This project will expand model-based entity resolution into the streaming data space through the formulation of new models and novel computational algorithms. Specifically, this project aims to improve scalability of Bayesian entity resolution models through approximate sampling techniques, such as variational inference, and develop fast updating of a Bayesian entity resolution model in a streaming data context, resulting in the first Bayesian entity resolution model updating strategy that can handle streaming data contexts for massive data sets. A limitation of existing methods for streaming inference with Bayesian models is that the pool of samples to be updated will converge to a degenerate distribution as the process is repeated many times, which guarantees poor quality of model fits in a streaming setting. This issue will be addressed with introduction of a novel Markov chain Monte Carlo sampler for streaming Bayesian inference, which will improve existing methods by combining filtering ideas with a highly parallelizable transition kernel. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.