Polynomial data compression for large-scale physics experiments

TL;DR

This paper introduces a lossless data compression algorithm tailored for large-scale physics experiments, achieving a balance between high compression rates and fast processing speeds, especially for astronomy and particle physics data.

Contribution

The paper presents a novel lossless compression algorithm optimized for physics data, demonstrating significant speed and efficiency improvements for large-scale experiments like CTA.

Findings

Fast and reasonably efficient standalone compression

Enhances existing methods like LZMA when used as pre-compression

Effective on real physics datasets from CTA

Abstract

The new generation research experiments will introduce huge data surge to a continuously increasing data production by current experiments. This data surge necessitates efficient compression techniques. These compression techniques must guarantee an optimum tradeoff between compression rate and the corresponding compression /decompression speed ratio without affecting the data integrity. This work presents a lossless compression algorithm to compress physics data generated by Astronomy, Astrophysics and Particle Physics experiments. The developed algorithms have been tuned and tested on a real use case~: the next generation ground-based high-energy gamma ray observatory, Cherenkov Telescope Array (CTA), requiring important compression performance. Stand-alone, the proposed compression method is very fast and reasonably efficient. Alternatively, applied as pre-compression algorithm, it can accelerate common methods like LZMA, keeping close performance.