Supercomputers as a Continous Medium

TL;DR

This paper introduces a homogeneous computer model for supercomputers that unifies various computational theories and reveals that current supercomputers are nearing fundamental physical performance limits.

Contribution

It proposes a continuous medium abstraction for supercomputers, linking application performance to physical system properties and recovering existing models from first principles.

Findings

Supercomputers are approaching fundamental physical limits.

The homogeneous model unifies existing computational theories.

Applications like CG and FFT are nearing these limits.

Abstract

As supercomputers' complexity has grown, the traditional boundaries between processor, memory, network, and accelerators have blurred, making a homogeneous computer model, in which the overall computer system is modeled as a continuous medium with homogeneously distributed computational power, memory, and data movement transfer capabilities, an intriguing and powerful abstraction. By applying a homogeneous computer model to algorithms with a given I/O complexity, we recover from first principles, other discrete computer models, such as the roofline model, parallel computing laws, such as Amdahl's and Gustafson's laws, and phenomenological observations, such as super-linear speedup. One of the homogeneous computer model's distinctive advantages is the capability of directly linking the performance limits of an application to the physical properties of a classical computer system. Applying the homogeneous computer model to supercomputers, such as Frontier, Fugaku, and the Nvidia DGX GH200, shows that applications, such as Conjugate Gradient (CG) and Fast Fourier Transforms (FFT), are rapidly approaching the fundamental classical computational limits, where the performance of even denser systems in terms of compute and memory are fundamentally limited by the speed of light.