Designing dedicated data compression for physics experiments within FPGA already used for data acquisition

TL;DR

This paper presents a specialized data compression system designed for FPGA-based data acquisition in physics experiments, significantly reducing data size by applying tailored encoding techniques to handle large volumes of detector data efficiently.

Contribution

The paper introduces a dedicated FPGA-compatible data compression method that leverages adaptive binning and channel-specific probability models to optimize data reduction in physics experiments.

Findings

Data compression reduces raw data size from 1170 to 437 bits per event.

Adaptive binning and channel-specific coding improve compression efficiency.

The proposed system fits within existing FPGA resources used for data acquisition.

Abstract

Physics experiments produce enormous amount of raw data, counted in petabytes per day. Hence, there is large effort to reduce this amount, mainly by using some filters. The situation can be improved by additionally applying some data compression techniques: removing redundancy and optimally encoding the actual information. Preferably, both filtering and data compression should fit in FPGA already used for data acquisition - reducing requirements of both data storage and networking architecture. We will briefly explain and discuss some basic techniques, for a better focus applied to design a dedicated data compression system basing on a sample data from a prototype of a tracking detector: 10000 events for 48 channels. We will focus on the time data here, which after neglecting the headers and applying data filtering, requires on average 1170 bits/event using the current coding. Encoding relative times (differences) and grouping data by channels, reduces this number to 798 bits/channel, still using fixed length coding: a fixed number of bits used for a given value. Using variable length Huffman coding to encode numbers of digital pulses for a channel and the most significant bits of values (simple binning) reduces further this number to 552 bits/event. Using adaptive binning: denser for frequent values, and an accurate entropy coder we get further down to 455 bits/event - this option can easily fit unused resources of FPGA currently used for data acquisition. Finally, using separate probability distributions for different channels, what could be done by a software compressor, leads to 437bits/event, what is 2.67 times less than the original 1170 bits/event.