SHF: Small: Programming Foundations for Real-Time Data Analysis

William Marsh Rice University, Houston TX

Investigators

Abstract

Modern information-processing systems handle large volumes of data that are produced in real time and need immediate analysis. These real-time data analyses are used to extract insights and enable decision-making in various application domains, such as business intelligence and patient monitoring. Existing systems for processing real-time (streaming) data focus on providing efficient and scalable execution over a distributed computing infrastructure. A common problem with these systems is that their behavior is not clearly specified, which makes it difficult to reason about their correctness. In many cases, the distributed execution of a streaming data analysis gives rise to unpredictable behavior that is not reproducible. The project's novelties are the development of a formal framework for specifying the behavior of real-time data-processing systems and the use of this framework for ensuring the reproducibility of computations. The project's impacts are advances in the practice of programming trustworthy and repeatable streaming data analyses. The project develops a novel programming foundation that encompasses the semantics of data-stream processing, the associated principles of query-language design, and a robust distributed implementation. A denotational semantics is described using a type discipline that captures various notions of data models and data invariants. Basic programming abstractions for stream processing are identified, which enable the modular specification of complex streaming analyses. An implementation is developed that guarantees efficient and reproducible distributed execution. The usefulness of the framework is established with case studies from the domains of enterprise business intelligence and patient monitoring in healthcare. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.