CAREER: Evolutionary Data Systems

Harvard University, Cambridge MA

Investigators

Abstract

We are entering the era of data deluge. Businesses, sciences and government organizations collect overwhelming amounts of data on a daily or even hourly basis. But data becomes useful information (knowledge) only after proper analysis. Data systems drive the analysis of data by servicing query requests. The way a system stores, accesses and processes data, defines the architecture of the data system. Over the past five decades the database research community and industry have designed numerous data systems architectures; a fundamental problem now is that there is no single data systems architecture that fits the ever-increasing kinds of data driven applications and scenarios. This is a critical problem for numerous reasons. First, anyone in need of a data system is confronted with numerous complex options, e.g., relational databases, NoSQL, NewSQL, column-stores, row-stores, and the list goes on. This overwhelming array of choices makes bootstrapping data-driven applications difficult and time consuming, requiring expertise often not accessible due to cost issues (e.g., to sciences or small businesses). Furthermore, many new kinds of data-driven applications are characterized by (a) varying workload patterns, i.e., varying patterns in data and queries and (b) new requirements in terms of interactive query processing and data exploration, i.e., when in need to quickly find useful knowledge in a big pile of data without a predefined exact goal. What these cases have in common is that one cannot tell a priori which data system architecture is the best fit as there is no predefined or fixed workload. An application may change over time and then a different architecture will be best. In short, this is a "one size does not fit all world" which sacrifices flexibility for performance. However, as sciences and businesses increase their ability to collect more data and thus create more data-driven applications and scenarios, this model does not scale. The goal of this proposal is to maintain the performance characteristics of the "one size does not fit all" world, while adding the flexibility to effortlessly support emerging applications and varying workloads. The researchers will study a new class of database architectures, evolutionary data systems, which evolve continuously to match the application's needs. The way data is stored and the way data is accessed changes to accommodate the data and query patterns. There is no need to make a priori complex set-up decisions, locking an application to a fixed system architecture that may soon become suboptimal due to workload changes. Instead, one starts using an evolutionary system simply by identifying the data. Then, as new data and queries come in, the system evolves such that its architecture matches the properties of the incoming workload. At any given point in time, an evolutionary system may employ multiple competing solutions down at the low level of database architectures such as using various combinations of data layouts, access methods and execution strategies. Then "the fittest wins" and becomes the dominant architecture until the environment (workload) changes. The researchers will demonstrate solutions that can seamlessly evolve (back and forth) between different architectures, i.e., from key-value stores to column-stores, from read-optimized to write-optimized systems and from structured to schema-free systems. In addition, the researchers will develop solutions for several key challenges: how systems evolve, management of overheads, how to extend an evolutionary system as well as comparison with brute force approaches. For further information see the project web site at: http://daslab.seas.harvard.edu/evosys