Evolutionary Data Systems

TL;DR

Evolutionary data systems dynamically adapt their architectures over time to changing workloads, reducing the need for expert tuning and enabling more flexible, cost-effective data management.

Contribution

This paper introduces the concept of evolutionary data systems that automatically evolve their architecture in response to workload changes, inspired by biological evolution.

Findings

Prototype demonstrates seamless evolution between key-value and column-store architectures.

Systems can adapt architecture based on workload properties over time.

Evolutionary approach reduces manual tuning and configuration efforts.

Abstract

Anyone in need of a data system today is confronted with numerous complex options in terms of system architectures, such as traditional relational databases, NoSQL and NewSQL solutions as well as several sub-categories like column-stores, row-stores etc. This overwhelming array of choices makes bootstrapping data-driven applications difficult and time consuming, requiring expertise often not accessible due to cost issues (e.g., to scientific labs or small businesses). In this paper, we present the vision of evolutionary data systems that free systems architects and application designers from the complex, cumbersome and expensive process of designing and tuning specialized data system architectures that fit only a single, static application scenario. Setting up an evolutionary system is as simple as identifying the data. As new data and queries come in, the system automatically evolves so that its architecture matches the properties of the incoming workload at all times. Inspired by the theory of evolution, at any given point in time, an evolutionary system may employ multiple competing solutions down at the low level of database architectures -- characterized as combinations of data layouts, access methods and execution strategies. Over time, "the fittest wins" and becomes the dominant architecture until the environment (workload) changes. In our initial prototype, we demonstrate solutions that can seamlessly evolve (back and forth) between a key-value store and a column-store architecture in order to adapt to changing workloads.