CRII: III: Improving the Utilization of Humans in Data Integration and Discovery

Worcester Polytechnic Institute, Worcester MA

Investigators

Abstract

New artificial intelligence (AI) solutions, especially Large Language Models (LLMs), are now widely adopted by the general public. ChatGPT, for example, has over a hundred million daily users. With this widespread adoption, people have begun questioning the role of humans in traditional data pipelines. Nevertheless, many still believe that human intelligence is irreplaceable. Humans can do things AI cannot such as understanding nuances and context and interpreting subjective or ambiguous situations. Thus, this project explores efficient ways to utilize human intelligence in data pipelines, especially with the emergence of LLMs. Among data pipelines, data integration and discovery are the cornerstones in contemporary data science, aiming at understanding datasets, extending and improving them, while focusing on the data. Humans play an integral role in these pipelines as data collectors, generators, and annotators. Accordingly, analyzing their involvement and investigating how to optimally utilize them is a necessity. Therefore, analyzing their involvement and investigating how to optimally utilize them is essential. As contemporary research in the field shifts towards employing LLMs, human involvement will need to be adapted accordingly. To this end, this project will introduce methods to support humans in data integration and discovery, lay the foundations for studying human involvement in these pipelines, and establish new methods to evaluate and benchmark them. This, in turn, also has the potential to create new human jobs, such as prompt engineers and response validators. This work will also contribute to the transparency of solutions via proper prompting and validation of AI responses. Finally, this work is expected to benefit society by facilitating responsible and open data science. Its solutions will be made publicly available along with high-quality benchmarks that will have scientific value for comparisons and settling debates, advancing this important field. This project will develop new data integration and discovery solutions that account for human-in-the-loop processes and the emergence of LLMs. Human-in-the-loop typically refers to leveraging human intelligence in data science pipelines. Human-in-the-loop data integration has received attention in the research literature, from pay-as-you-go frameworks to crowdsourcing, with a common understanding that it requires domain expertise. In contrast, the explicit role of humans in data discovery has yet to be thoroughly explored. Implicitly, humans are consistently involved as data collectors, generators, and annotators. Therefore, the vision of this project is to effectively utilize human intelligence in data integration and discovery as the field increasingly employs LLMs. The research methodology builds upon, integrates, and extends work on scalable data integration and discovery; cognitive and meta-cognitive psychology; and interactions with large language models. This project will investigate fundamental questions about the efficient utilization of humans in data integration and discovery. Specifically, it will address the following research challenges: 1) Understanding the involvement of humans in contemporary data integration and discovery pipelines; 2) Uncovering human biases that interfere with or benefit these processes; 3) Designing future solutions to efficiently involve humans in data integration and discovery, especially with the rise of LLMs. Throughout the development, this project will also develop new evaluation frameworks that consider how humans interact with the data. A key motivation for data discovery is that common benchmarks are often created manually, without accounting for the biases introduced by their creators. This project aims to clean and improve such benchmarks, making them publicly available. Additionally, involving humans in-the-loop will enhance users' trust in the data and the outputs generated by LLMs. By understanding AI-generated solutions through prompting and validating responses, users can better utilize these solutions in their data science tasks. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.