Streaming Data in HPC Workflows Using ADIOS

TL;DR

This paper introduces the Sustainable Staging Transport (SST), an ADIOS engine enabling direct data streaming in HPC workflows to overcome the IO Wall problem, improving performance and usability without requiring source code modifications.

Contribution

The paper presents SST, a novel ADIOS engine that facilitates direct streaming between applications, enhancing data transfer efficiency in HPC workflows without source code changes.

Findings

SST achieves higher bandwidth than filesystem limits in model training workflows.

SST enables strong coupling of applications for multiphysics simulations.

SST supports in situ analysis and visualization with improved data transfer performance.

Abstract

The "IO Wall" problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS "engine", accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier's file system, for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.