Scalable In Situ Compression of Transient Simulation Data Using Time-Dependent Bases

TL;DR

This paper introduces a scalable in situ data compression method for large-scale time-dependent simulations, using time-dependent bases to significantly reduce storage needs while maintaining low reconstruction error.

Contribution

It develops a novel in situ compression technique based on time-dependent subspaces that scales linearly with data size and does not require data history, enabling efficient handling of streaming simulation data.

Findings

Achieves orders of magnitude data reduction

Maintains reconstruction error below threshold

Demonstrated on diverse flow simulation cases

Abstract

Large-scale simulations of time-dependent problems generate a massive amount of data and with the explosive increase in computational resources the size of the data generated by these simulations has increased significantly. This has imposed severe limitations on the amount of data that can be stored and has elevated the issue of input/output (I/O) into one of the major bottlenecks of high-performance computing. In this work, we present an in situ compression technique to reduce the size of the data storage by orders of magnitude. This methodology is based on time-dependent subspaces and it extracts low-rank structures from multidimensional streaming data by decomposing the data into a set of time-dependent bases and a core tensor. We derive closed-form evolution equations for the core tensor as well as the time-dependent bases. The presented methodology does not require the data history and the computational cost of its extractions scales linearly with the size of data -- making it suitable for large-scale streaming datasets. To control the compression error, we present an adaptive strategy to add/remove modes to maintain the reconstruction error below a given threshold. We present four demonstration cases: (i) analytical example, (ii) incompressible unsteady reactive flow, (iii) stochastic turbulent reactive flow, and (iv) three-dimensional turbulent channel flow.