Pangea: Monolithic Distributed Storage for Data Analytics

TL;DR

Pangea is a unified monolithic storage system designed for data analytics that simplifies data management, reduces redundancy, and improves performance compared to layered systems like Spark and HDFS.

Contribution

It introduces a single, integrated storage system that manages all data types and operations without layering, enhancing efficiency and simplifying data management.

Findings

Pangea's performance is comparable or superior to layered systems like Spark.

Reduces data copying and management overhead.

Improves resource utilization in data analytics workloads.

Abstract

Storage and memory systems for modern data analytics are heavily layered, managing shared persistent data, cached data, and non-shared execution data in separate systems such as distributed file system like HDFS, in-memory file system like Alluxio and computation framework like Spark. Such layering introduces significant performance and management costs for copying data across layers redundantly and deciding proper resource allocation for all layers. In this paper we propose a single system called Pangea that can manage all data---both intermediate and long-lived data, and their buffer/caching, data placement optimization, and failure recovery---all in one monolithic storage system, without any layering. We present a detailed performance evaluation of Pangea and show that its performance compares favorably with several widely used layered systems such as Spark.