Easy Acceleration with Distributed Arrays

TL;DR

This paper demonstrates that distributed arrays enable scalable high-performance computing across diverse hardware, achieving near-linear scaling and massive bandwidth on supercomputing infrastructure.

Contribution

It provides an empirical evaluation of distributed array performance across multiple hardware generations, highlighting scalability and hardware improvements over time.

Findings

Horizontal scaling across nodes was linear.

Achieved over 1 PB/s bandwidth on supercomputing infrastructure.

Documented 10x, 100x, and 5x increases in memory bandwidth over 20 years.

Abstract

High level programming languages and GPU accelerators are powerful enablers for a wide range of applications. Achieving scalable vertical (within a compute node), horizontal (across compute nodes), and temporal (over different generations of hardware) performance while retaining productivity requires effective abstractions. Distributed arrays are one such abstraction that enables high level programming to achieve highly scalable performance. Distributed arrays achieve this performance by deriving parallelism from data locality, which naturally leads to high memory bandwidth efficiency. This paper explores distributed array performance using the STREAM memory bandwidth benchmark on a variety of hardware. Scalable performance is demonstrated within and across CPU cores, CPU nodes, and GPU nodes. Horizontal scaling across multiple nodes was linear. The hardware used spans decades and allows a direct comparison of hardware improvements for memory bandwidth over this time range; showing a 10x increase in CPU core bandwidth over 20 years, 100x increase in CPU node bandwidth over 20 years, and 5x increase in GPU node bandwidth over 5 years. Running on hundreds of MIT SuperCloud nodes simultaneously achieved a sustained bandwidth $>$1 PB/s.