III: Small: Semantic Version Management in Data Lakes

Northeastern University, Boston MA

Investigators

Abstract

Data fuels our economy. Those who work with data invest their efforts in finding effective ways to extract knowledge and to handle its increasing size. This growth is not only characterized by new sources of data, but also by data replication - that is, the copying, integration, and modification of datasets that creates new versions of datasets. Notably, consultants such as the International Data Corporation estimate that most of the newly generated data being used in business are versions of existing data. Understanding data therefore requires a semantic understanding of data versioning, which becomes a key ingredient in handling and managing data. This project will focus on advancing the scientific understanding of data versioning and will fundamentally contribute to any science or activity that uses data, which nowadays covers a tremendous amount of all human activity. To advance open data science, this project will lay the foundations for semantic understanding of data changes that result in new versions of data and will introduce scalable tools to uncover and explain data changes. This will contribute both to the development of effective frameworks to handle multiple data versions within a data science pipeline as well as to the design of systems that incorporate and manage data replication. This work is expected to also benefit society by facilitating responsible and open data science. Its solutions will be made publicly available and provided alongside high-quality highly curated benchmarks that themselves will have scientific value in allowing comparisons and settling scientific debates in order to advance this important field. The project will also use aspects of "responsible data science", aiming to ensure fairness, accuracy, confidentiality, and transparency when working with data. This project will develop a new paradigm we call semantic version management. The vision is to enable users, with minimal upfront effort, to understand the multitude of versions that typically reside in data lakes. The main objective of the project is to enable data scientists who currently rely mainly on file names to find the "right" version of a dataset to see and understand the changes (cleaning, value imputation, integration, and others) that have been made between datasets. The research methodology builds-on, integrates, and extends work on scalable data discovery; program by example and data transformation synthesis; and learning schema mappings from inconsistent and incomplete evidence. This project will develop methods to support the semantic understanding of data versioning, lay the foundations for studying data versions, and establish new methods for evaluating and benchmarking data versioning. Specifically, this project will address the following fundamental research challenges: 1) recovering transformations done to data and explain how one dataset differs from another version of the dataset; 2) efficiently finding versions of a dataset from within a massive table repository or data lake; and 3) understanding the version history among a collection of versions and constructing a graph that expresses the story behind the creation of the data versions. Throughout the development, this project will also develop new evaluation frameworks that not only consider the correctness of solutions, but also their explainability. An important motivation for semantic version management is to give users more trust in the data they are using. If they understand the transformations used to derive one version from another, they can better understand if a version meets the needs of their data science task. In addition, new benchmarks will be generated and shared with the community to encourage open science and allow reliable comparison with new or alternative approaches to version understanding. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.