CRII: III: Statistical Learning and Inference Methods for Automated Data Cleaning

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

The recent investments in information retrieval, natural language processing, and AI have enabled computers to interpret what they see; read and analyze unstructured data; answer complex questions; and interact with their environment. There is a hidden catch, however: the reliance of all these state-of-the-art systems on high-effort tasks like data preparation and data cleaning. It is estimated that 70% to 80% percent of the time devoted on analytics projects is spent on checking and organizing data. The challenge is that data collection often introduces incomplete, erroneous, replicated, or conflicting data records. The burden of data preparation has led to many efforts in automating isolated tasks related to data cleaning, such as record de-duplication. However, success with end-to-end data cleaning has been limited, especially in the presence of critical data driven applications. Here, human engagement is normally required to guide and evaluate the impact of data cleaning. This project investigates the design of partly-automated, interactive data cleaning systems that are efficient for large-scale applications and come with formal accuracy guarantees. The emphasis of this work is on data cleaning methods that combine human expertise with statistical learning and probabilistic inference to model the inherent noise of raw data; and repair incomplete, inconsistent or erroneous records. The main hypothesis driving this work is that statistical learning allows us to reason about heterogeneous signals that are indicative of the correct latent value of a data record. This project will develop a formal statistical framework for data cleaning and weakly supervised machine learning solutions for interactive data cleaning over structured or semi-structured data. The outcomes of this project will have the power to significantly ease the currently challenging procedure of manually inspecting data to be used in downstream analytical tasks. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.