Collaborative Research: Chameleon: A Large-Scale, Reconfigurable Experimental Environment for Cloud Research

University Of Chicago, Chicago IL

Investigators

Abstract

A persistent problem facing academic cloud research is the lack of infrastructure and data to perform experimental research: large-scale hardware is needed to investigate the scalability of cloud infrastructure and applications, heterogeneous hardware is needed to investigate algorithmic and implementation tradeoffs, fully-configurable software environments are needed to investigate the performance of virtualization techniques and the differences between cloud software stacks, and data about how clouds are used is needed to evaluate virtual machine scheduling and data placement algorithms. The Chameleon project will addresses these needs by providing a large-scale, fully configurable experimental testbed driven by the needs of the cloud research and education communities. The testbed, and the ecosystem associated with it, will enable researchers to explore a range of cloud research challenges, from large scale to small scale, including exploring low-level problems in hardware architecture, systems research, network configuration, and software design, or at higher levels of abstraction looking at cloud scheduling, cloud platforms, and cloud applications. Chameleon will significantly enhance the ability of the computing research community to understand the behavior of Internet scale cloud systems, and to develop new software, ideas and algorithms for the cloud environment. As the tremendous shift to cloud as the primary means of providing computing infrastructure continues, a large-scale testbed tailored to researchers' needs is essential to the continued relevance of a large fraction of computing research. The project is led by the University of Chicago and includes partners from the Texas Advanced Computing Center (TACC), Northwestern University, the Ohio State University, and the University of Texas at San Antonio, comprising a highly qualified and experienced team, with research leaders from the cloud and networking world blended with providers of production quality cyberinfrastructure. The team includes members from the NSF-supported FutureGrid project and from the GENI community, both forerunners of the NSFCloud solicitation under which this project is funded. The Chameleon testbed, will be deployed at the University of Chicago (UC) and the Texas Advanced Computing Center (TACC) and will consist of 650 multi-core cloud nodes, 5PB of total disk space, and leverage 100 Gbps connection between the sites. While a large part of the testbed will consist of homogenous hardware to support large-scale experiments, a portion of it will support heterogeneous units allowing experimentation with high-memory, large-disk, low-power, GPU, and co-processor units. The project will also leverage existing FutureGrid hardware at UC and TACC in its first year to provide a transition period for the existing FutureGrid community of experimental users. To support a broad range of experiments emphasizing a range of requirements ranging from a high degree of control to ease of use the project will support a graduated configuration system allowing full user configurability of the stack, from provisioning of bare metal and network interconnects to delivery of fully functioning cloud environments. In addition, to facilitate experiments, Chameleon will support a set of services designed to meet researchers needs, including support for experimental management, reproducibility, and repositories of trace and workload data of production cloud workloads. To facilitate the latter, the project will form a set of partnerships with commercial as well as academic clouds, such as Rackspace and Open Science Data Cloud (OSDC). It will also partner with other testbeds, notably GENI and INRIA's Grid5000 testbed, and reach out to the user community to shape the policy an direction of the testbed. The Chameleon project will bring a new dimension and scale of resources to the CS community who wish to educate their students about design, implementation, operation and applications of cloud computing, a critical skillset for future computing professionals. It will enhance the understanding and application of experimental methodology in computer science and generate new educational materials and resources, with the participation of, and for, Minority Serving Institution (MSI) students.