Disaggregated Database Management Systems

TL;DR

Disaggregated DBMSs leverage cloud infrastructure to separate processing and storage, enabling elastic scaling and dynamic workload optimization, but face technical challenges and opportunities in hardware-software co-design.

Contribution

This paper summarizes a panel discussion on the capabilities, challenges, and opportunities of disaggregated database management systems and hardware systems.

Findings

Disaggregated DBMSs enable elastic scaling of compute and storage.

Workload-aware dynamic assembly improves efficiency and cost-effectiveness.

Disaggregation introduces technical challenges in hardware-software integration.

Abstract

Modern applications demand high performance and cost efficient database management systems (DBMSs). Their workloads may be diverse, ranging from online transaction processing to analytics and decision support. The cloud infrastructure enables disaggregation of monolithic DBMSs into components that facilitate software-hardware co-design. This is realized using pools of hardware resources, i.e., CPUs, GPUs, memory, FPGA, NVM, etc., connected using high-speed networks. This disaggregation trend is being adopted by cloud DBMSs because hardware re-provisioning can be achieved by simply invoking software APIs. Disaggregated DBMSs separate processing from storage, enabling each to scale elastically and independently. They may disaggregate compute usage based on functionality, e.g., compute needed for writes from compute needed for queries and compute needed for compaction. They may also use disaggregated memory, e.g., for intermediate results in a shuffle or for remote caching. The DBMS monitors the characteristics of a workload and dynamically assembles its components that are most efficient and cost effective for the workload. This paper is a summary of a panel session that discussed the capability, challenges, and opportunities of these emerging DBMSs and disaggregated hardware systems.