Inference-Sufficient Representations for High-Throughput Measurement: Lessons from Lossless Compression Benchmarks in 4D-STEM

TL;DR

This study benchmarks various lossless compression methods for 4D-STEM datasets, revealing that optimized algorithms can significantly reduce data size and transfer times, but highlight the need for inference-driven data representations for sustainable high-throughput workflows.

Contribution

The paper systematically compares lossless compression techniques for 4D-STEM data, providing practical guidance and emphasizing the importance of inference-sufficient representations over raw data storage.

Findings

Blosc extunderscore zstd achieves comparable compression to gzip-9 but is much faster.

Compression ratios are highly reproducible and follow a power law with data sparsity.

4D-STEM data can be routinely compressed by over 10 times.

Abstract

Four-dimensional scanning transmission electron microscopy (4D-STEM) generates multi-gigabyte datasets, creating a growing mismatch between acquisition rates and practical storage, transfer, and interactive visualization capabilities. We systematically benchmark 13 lossless compression implementations across 5 representative datasets (8~MiB to 8~GiB, 49.5--92.8\% sparsity), with 10 independent runs per method. HDF5 provides built-in gzip compression, of which gzip-9 typically achieves the highest compression ratio but is slow. We therefore evaluate widely available alternatives (via hdf5plugin), including the Blosc family. As a representative comparison, blosc\_zstd achieves compression comparable to gzip-9 (mean 13.5$\times$ vs 12.3$\times$) while compressing 19--69$\times$ faster and reading 1.9--2.6$\times$ faster across datasets. Compression ratios are deterministic, and timing measurements are highly reproducible (CV $<$2\%). Compression performance follows a power law with sparsity ($R^2 = 0.99$), ranging from 5$\times$ for moderately sparse data to 35$\times$ for highly sparse data. We identify six top-performing implementations optimized for different use cases and demonstrate that 4D-STEM data can be routinely compressed by $>$10$\times$. While these results provide practical guidance for lossless compression selection, the broader conclusion is that lossless compression preserves measurements but does not by itself guarantee sustainable high-throughput workflows. As detector rates rise, data handling will increasingly require inference-driven representations -- i.e., deciding what must be preserved to support a scientific inference, rather than defaulting to storing fully dense raw measurements.