CAREER: Intermittent Query Processing

University Of Chicago, Chicago IL

Investigators

Abstract

Database systems that facilitate the storage and analysis of data are the backbone of modern data-intensive applications, from machine learning to the Internet of Things, and represent a significant market force with an estimated worldwide market value of over 50 billion USD. Many new database applications demand fast complex analytics over changing data. However, the slowdown in the growth of hardware performance, coupled with the continued increase of data and data-intensive applications creates a crisis, as existing database systems are not well suited for applications that want to trade-off performance and resource efficiency. To address this gap of queries over fresh data with controlled resource utilization, a new database system paradigm is needed. This award investigates a new query execution paradigm, Intermittent Query Processing, to answer the question: how should database systems provide low latency for ongoing queries with bursty data arrival or periodic queries while minimizing resource utilization? This framework allows a query to initially execute as a batch, with subsequent intermittent processing to update the result while minimizing the resources required to update the query result by selectively persisting artifacts from the initial query execution and by exploiting knowledge about expected data arrivals and models for how amenable a query is for incremental computation. This project will investigate methods for predicting query and data arrivals, scheduling and placement decisions for workloads where queries partially overlap, opportunities for state sharing, resource allocation, and new query optimization techniques. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.