CAREER: Understanding the Performance of Distributed Systems through Causal Tracing

Brown University, Providence RI

Investigators

Abstract

CAREER: Understanding the Performance of Distributed Systems through Causal Tracing Society increasingly depends on shared software systems that are large, decentralized, with many components that interact in complex and subtle ways. These systems include financial and banking services, Web and cloud resources, airline reservations, and big data and scientific computing, to name a few. Despite their unquestionable reach and success, in these systems it is very hard to answer questions about the causes of failures, to uncover dependency issues among their components, to determine the impact of one operation on the rest of the system, or to provide guarantees about their performance to users. By developing and applying techniques to enable deep and real-time understanding of the performance characteristics of large-scale distributed systems, this project?s goal is to develop techniques that will enable users and providers of these systems to better express their needs and their guarantees in terms of performance, and better plan for and mitigate the effects of failures. The main insight in this project is that because of the many components in distributed systems, the context of an operation initiated in one component gets lost as the operation involves other components. This makes it hard for a component deep in the system to discern with which client it is working, making it also hard to apply consistent policies or account for the cost of operations across component boundaries. This research will create the abstraction of a Tracing Plane that preserves this context throughout the entire execution of the system, allowing for debugging and diagnosis of performance problems, and for real-time provisioning of performance guarantees. This Tracing Plane will be a pervasive infrastructure to collect causal information from the execution of a distributed system and facilitate the efficient deployment of analytics and diagnostic tasks. Further, by aggregating information about tasks in the system across all components in a coherent way, the Tracing Plane enables the implementation of resource management policies that can act locally, in real-time, and with global knowledge - which is presently not possible. We are better today at building large-scale distributed systems than we are at understanding precisely how they work, and how they fail and this will provide a core educational aspect, as the Tracing Plane is a strong pedagogical tool for the understanding of distributed systems structure and execution. This work will engage undergraduate and graduate students, as well as industry partners that operate such large-scale distributed systems. By starting from increased visibility into these systems, the ultimate goal of this project is to provide tools and methods to allow building, operation, and management of large-scale, shared distributed systems that are efficient, reliable, and predictable. As society increasingly depends on systems of this kind, this research has a large and long lasting potential impact.