Exoshuffle: An Extensible Shuffle Architecture

TL;DR

Exoshuffle introduces an extensible, flexible shuffle architecture built on Ray, achieving high performance and scalability while enabling new ML applications and simplifying the implementation of shuffle optimizations.

Contribution

It presents a novel, decoupled shuffle architecture that is more flexible and easier to extend than monolithic systems, with competitive performance and scalability.

Findings

Achieved shuffle performance comparable to monolithic systems.

Set a new record as the most cost-efficient sorting system.

Enabled ML training applications to leverage scalable shuffle easily.

Abstract

Shuffle is one of the most expensive communication primitives in distributed data processing and is difficult to scale. Prior work addresses the scalability challenges of shuffle by building monolithic shuffle systems. These systems are costly to develop, and they are tightly integrated with batch processing frameworks that offer only high-level APIs such as SQL. New applications, such as ML training, require more flexibility and finer-grained interoperability with shuffle. They are often unable to leverage existing shuffle optimizations. We propose an extensible shuffle architecture. We present Exoshuffle, a library for distributed shuffle that offers competitive performance and scalability as well as greater flexibility than monolithic shuffle systems. We design an architecture that decouples the shuffle control plane from the data plane without sacrificing performance. We build Exoshuffle on Ray, a distributed futures system for data and ML applications, and demonstrate that we can: (1) rewrite previous shuffle optimizations as application-level libraries with an order of magnitude less code, (2) achieve shuffle performance and scalability competitive with monolithic shuffle systems, and break the CloudSort record as the world's most cost-efficient sorting system, and (3) enable new applications such as ML training to easily leverage scalable shuffle.