CRII: CSR: An Asynchronous Design to Reduce the Long Tail Latency of n-Tier Applications in the Cloud

Louisiana State University, Baton Rouge LA

Investigators

Abstract

Low latency is essential for web facing e-commerce applications. For example, Amazon reported that every 100ms increase in the page load time correlates to a decrease in sales by 1%. However, practitioners have consistently experienced the long tail latency problem, or wide response time variations, of web applications in the cloud as the utilization of the system reaches moderate levels (e.g., 50%). A practical consequence is that enterprise cloud data centers have been reported to purposefully keep utilization levels low (e.g., 18%) to avoid the penalties caused by the long tail latency problem, wasting considerable computing resources and power. Solving the latency long tail problem will have a great impact on achieving higher cost efficiency of cloud computing resources, leading to lower cost of cloud users and higher return of cloud providers, among other benefits of more sustainable large-scale computing infrastructures. The main goal of this project is to investigate and resolve a key factor that causes the long tail latency problem of n-tier web applications: the Cross-Tier Queue Propagation. The Cross-Tier Queue Propagation is an effect caused by strong inter-dependencies in the request processing chain of an n-tier web application with synchronous inter-tier communication (RPC-style call/response). With such Cross-Tier Queue Propagation, a small queuing delay in a downstream server (e.g., a database) can be propagated and amplified to upstream tiers (e.g., a web server), leading to wide response time variations. This research proposes an asynchronous design of inter-tier communication that addresses the long tail latency problem caused by the Cross-Tier Queue Propagation of an n-tier system. The rationale of asynchronous communication is to decouple complex inter-tier dependencies enforced by traditional RPC-style synchronous communication. Servers involved in asynchronous communication adopt an event-driven design in which the processing threads in one component server are "independent" from those in other component servers in the request processing chain. The new design is expected to break the Cross-Tier Queue Propagation in the request processing chain and prevent the amplification of small queuing delay into the long tail latency problem. To validate the hypothesis, this research will design a representative asynchronous 3-tier system consisting of component servers that support asynchronous communication, implement asynchronous benchmark applications that are compatible with the developed asynchronous servers, and run large-scale cloud experiments and validate the effectiveness of the asynchronous design to break the Cross-Tier Queue Propagation.