Fast Entropy Coding for ALICE Run 3

TL;DR

This paper introduces a fast, resource-efficient entropy coding scheme using rANS for real-time data compression in the ALICE Run 3 experiment, achieving high compression rates within strict timing constraints.

Contribution

It presents a novel multi-process architecture for entropy coding with extensions to rANS, optimized for large, sparse alphabets and real-time data processing in high-energy physics.

Findings

Achieves real-time compression at 50 kHz data rate

Uses static distribution tables for resource efficiency

Handles large, sparse source alphabets up to 25 bits per symbol

Abstract

In LHC Run 3, the upgraded ALICE detector will record Pb-Pb collisions at a rate of 50 kHz usingcontinuous readout. The resulting stream of raw data at 3.5 TB/s has to be processed with a setof lossy and lossless compression and data reduction techniques to a storage data rate of 90 GB/swhile preserving relevant data for physics analysis. This contribution presents a custom losslessdata compression scheme based on entropy coding as the final component in the data reductionchain which has to compress the data rate from 300 GB/s to 90 GB/s. A flexible, multi-processarchitecture for the data compression scheme is proposed that seamlessly interfaces with the datareduction algorithms of earlier stages and allows to use parallel processing in order to keep therequired firm real-time guarantees of the system. The data processed inside the compressionprocess have a structure that allows the use of an rANS entropy coder with more resource efficientstatic distribution tables. Extensions to the rANS entropy coder are introduced to efficientlywork with these static distribution tables and large but sparse source alphabets consisting of upto 25 Bit per symbol. Preliminary performance results show compliance with the firm real-timerequirements while offering close-to-optimal data compression.