LightAMR format standard and lossless compression algorithms for adaptive mesh refinement grids: RAMSES use case

TL;DR

This paper introduces the lightAMR data model and lossless compression algorithms for adaptive mesh refinement grids, significantly reducing data size and redundancy in RAMSES simulation datasets to improve I/O efficiency.

Contribution

It proposes a lightweight post-processing data model for AMR meshes, along with pruning and compression algorithms, enhancing data compactness and interoperability.

Findings

Pruning reduces cell count by 10-40% without data loss.

Grid structure can be compressed by ~1000x.

Scalar fields compressed by 1.2 to 1.5 times.

Abstract

The evolution of parallel I/O library as well as new concepts such as 'in transit' and 'in situ' visualization and analysis have been identified as key technologies to circumvent I/O bottleneck in pre-exascale applications. Nevertheless, data structure and data format can also be improved for both reducing I/O volume and improving data interoperability between data producer and data consumer. In this paper, we propose a very lightweight and purpose-specific post-processing data model for AMR meshes, called lightAMR. Based on this data model, we introduce a tree pruning algorithm that removes data redundancy from a fully threaded AMR octree. In addition, we present two lossless compression algorithms, one for the AMR grid structure description and one for AMR double/single precision physical quantity scalar fields. Then we present performance benchmarks on RAMSES simulation datasets of this new lightAMR data model and the pruning and compression algorithms. We show that our pruning algorithm can reduce the total number of cells from RAMSES AMR datasets by 10-40% without loss of information. Finally, we show that the RAMSES AMR grid structure can be compacted by ~ 3 orders of magnitude and the float scalar fields can be compressed by a factor ~ 1.2 for double precision and ~ 1.3 - 1.5 in single precision with a compression speed of ~ 1 GB/s.