Beyond 16GB: Out-of-Core Stencil Computations

TL;DR

This paper presents techniques for out-of-core stencil computations that enable solving three times larger problems than available fast memory, with minimal efficiency loss, on modern high-performance architectures.

Contribution

It introduces cache-blocking tiling optimizations for large-scale stencil codes using the OPS language, addressing memory limitations in high-performance computing.

Findings

Able to solve 3x larger problems than fast memory size

Achieved at most 15% efficiency loss

Validated on Intel Knights Landing and NVIDIA P100 GPUs

Abstract

Stencil computations are a key class of applications, widely used in the scientific computing community, and a class that has particularly benefited from performance improvements on architectures with high memory bandwidth. Unfortunately, such architectures come with a limited amount of fast memory, which is limiting the size of the problems that can be efficiently solved. In this paper, we address this challenge by applying the well-known cache-blocking tiling technique to large scale stencil codes implemented using the OPS domain specific language, such as CloverLeaf 2D, CloverLeaf 3D, and OpenSBLI. We introduce a number of techniques and optimisations to help manage data resident in fast memory, and minimise data movement. Evaluating our work on Intel's Knights Landing Platform as well as NVIDIA P100 GPUs, we demonstrate that it is possible to solve 3 times larger problems than the on-chip memory size with at most 15\% loss in efficiency