CAREER: Towards Scalable Datacenter Services with Strong Robustness Guarantees

Princeton University, Princeton NJ

Investigators

Abstract

Modern datacenters have reached hundreds of thousands of servers, and modern distributed services can be spread over multiple or even dozens of datacenters worldwide. For scalability, availability, and performance, these services have increasingly embraced a weakened model of data consistency. This trade-off proved highly successful for applications such as web crawling, search, and content distribution. On the other hand, the recent trend to move ever-more dynamic applications to the "cloud" portends a shift in service requirements---in which losing data or applying operations out-of-order may not be acceptable---as does far-flung demand for concurrent access to data and services. To meet these changing needs, this research project reconsiders the challenge of building storage and replicated systems with strong robustness guarantees and at scale. Recognizing that large-scale, complex systems are typically built-up from smaller groups---which either provide different functionality or partition some larger state-space into smaller, more manageable parts---it leverages a group compositional approach to tackle the problem. First, the research develops novel protocols for smaller groups of nodes that offer strong properties at minimal overhead. Second, it proposes a coordination service and a suite of management algorithms that adaptively organizes these groups and composes them together. These algorithms explore problems of dynamic load balancing, topological control, and security. The research includes substantial systems building, including the incorporation of its core services into several other distributed systems (e.g., a scalable virtual world and a istributed name resolution service). The remarkably rich and varied Internet services run out of datacenters form the core of people's online experiences today. The algorithms and software systems developed by this research may lower the barrier to developing datacenter services that are scalable, reliable, secure, efficient, and easy to manage. And by reducing the technical difficulty of building robust, large-scale applications---by providing them with a firm foundation---this research may enable further innovation in Internet services.