Online randomized interpolative decomposition with a posteriori error estimator for temporal PDE data reduction

TL;DR

This paper introduces an online randomized interpolative decomposition algorithm with a posteriori error estimation for efficient, in situ data reduction of large-scale PDE simulation data, enabling real-time analysis and improved interpretability.

Contribution

The paper presents a novel streaming ID algorithm using ridge leverage scores and Hutch++ error estimator, reducing computational passes and enhancing real-time PDE data analysis.

Findings

Single-pass algorithm effectively reduces data size in PDE simulations.

Real-time error estimation guides optimal data approximation.

Numerical experiments demonstrate superior performance over offline methods.

Abstract

Traditional low-rank approximation is a powerful tool to compress the huge data matrices that arise in simulations of partial differential equations (PDE), but suffers from high computational cost and requires several passes over the PDE data. The compressed data may also lack interpretability thus making it difficult to identify feature patterns from the original data. To address these issues, we present an online randomized algorithm to compute the interpolative decomposition (ID) of large-scale data matrices {\em in situ}. Compared to previous randomized IDs that used the QR decomposition to determine the column basis, we adopt a streaming ridge leverage score-based column subset selection algorithm that dynamically selects proper basis columns from the data and thus avoids an extra pass over the data to compute the coefficient matrix of the ID. In particular, we adopt a single-pass error estimator based on the non-adaptive Hutch++ algorithm to provide real-time error approximation for determining the best coefficients. As a result, our approach only needs a single pass over the original data and thus is suitable for large and high-dimensional matrices stored outside of core memory or generated in PDE simulations. A strategy to improve the accuracy of the reconstructed data gradient, when desired, within the ID framework is also presented. We provide numerical experiments on turbulent channel flow and ignition simulations, and on the NSTX Gas Puff Image dataset, comparing our algorithm with the offline ID algorithm to demonstrate its utility in real-world applications.