Distributed Ranges: A Model for Distributed Data Structures, Algorithms, and Views

TL;DR

This paper introduces distributed ranges, a C++ model for building interoperable, high-level distributed data structures and algorithms that efficiently operate across multiple memory locales, enabling flexible parallel programming.

Contribution

The paper presents a novel distributed ranges model that extends C++ ranges with segmentation, facilitating interoperability and composability of distributed data structures, views, and algorithms.

Findings

Distributed ranges enable interoperability across different libraries.

High-level algorithms using distributed ranges achieve competitive performance.

The model supports recursive composition of views and algorithms.

Abstract

Data structures and algorithms are essential building blocks for programs, and \emph{distributed data structures}, which automatically partition data across multiple memory locales, are essential to writing high-level parallel programs. While many projects have designed and implemented C++ distributed data structures and algorithms, there has not been widespread adoption of an interoperable model allowing algorithms and data structures from different libraries to work together. This paper introduces distributed ranges, which is a model for building generic data structures, views, and algorithms. A distributed range extends a C++ range, which is an iterable sequence of values, with a concept of segmentation, thus exposing how the distributed range is partitioned over multiple memory locales. Distributed data structures provide this distributed range interface, which allows them to be used with a collection of generic algorithms implemented using the distributed range interface. The modular nature of the model allows for the straightforward implementation of \textit{distributed views}, which are lightweight objects that provide a lazily evaluated view of another range. Views can be composed together recursively and combined with algorithms to implement computational kernels using efficient, flexible, and high-level standard C++ primitives. We evaluate the distributed ranges model by implementing a set of standard concepts and views as well as two execution runtimes, a multi-node, MPI-based runtime and a single-process, multi-GPU runtime. We demonstrate that high-level algorithms implemented using generic, high-level distributed ranges can achieve performance competitive with highly-tuned, expert-written code.