An octree-based sampling algorithm for analyzing big simulation data

TL;DR

This paper introduces an improved octree-based sampling algorithm, S^3, that efficiently reduces large CFD datasets by up to 98%, enabling detailed analysis on standard workstations.

Contribution

An enhanced S^3 algorithm that generates a time-invariant octree grid for effective data down-sampling in large-scale flow simulations.

Findings

Reduces data size by 35% to 98% across various flow cases.

Preserves dominant flow dynamics during sampling.

Enables post-processing on local workstations instead of HPC resources.

Abstract

As computational resources continue to increase, the storage and analysis of vast amounts of data will inevitably become a bottleneck in computational fluid dynamics (CFD) and related fields. Although compression algorithms and efficient data formats can mitigate this issue, they are often insufficient when post-processing large amounts of volume data. Processing such data may require additional high-performance software and resources, or it may restrict the analysis to shorter time series or smaller regions of interest. The present work proposes an improved version of the existing \emph{Sparse Spatial Sampling} algorithm ($S^3$) to reduce the data from time-dependent flow simulations. The $S^3$ algorithm iteratively generates a time-invariant octree grid based on a user-defined metric, efficiently down-sampling the data while aiming to preserve as much of the metric as possible. Using the sampled grid allows for more efficient post-processing and enables memory-intensive tasks, such as computing the modal decomposition of flow snapshots. The enhanced version of $S^3$ is tested and evaluated on the scale-resolving simulations of the flow past a tandem configuration of airfoils in the transonic regime, the incompressible turbulent flow past a circular cylinder, and the flow around an aircraft half-model at high Reynolds and Mach numbers. $S^3$ significantly reduces the number of mesh cells by $35 \%$ to $98\%$ for all test cases while accurately preserving the dominant flow dynamics, enabling post-processing of CFD data on a local workstation rather than HPC resources for many cases.