Real-time data compression for data acquisition systems applied to the ITER Radial Neutron Camera

TL;DR

This paper presents a real-time, parallel data compression system using LZ4 for the ITER Radial Neutron Camera, achieving significant data reduction and high-speed processing in a high-throughput data acquisition environment.

Contribution

It introduces a novel parallel implementation of LZ4 compression tailored for high-speed data acquisition in fusion diagnostics.

Findings

Achieved up to 50% data size reduction.

Maintained compression speeds up to 400 MB/s per core.

Demonstrated system stability over 30-minute long acquisitions.

Abstract

To achieve the aim of the ITER Radial Neutron Camera Diagnostic, the data acquisition prototype must be compliant with a sustained 2 MHz peak event for each channel with 128 samples of 16 bits per event. The data is acquired and processed using an IPFN FPGA Mezzanine Card (FMC-AD2-1600) with 2 digitizer channels of 12-bit resolution and a sampling rate up to 1.6 GSamples/s mounted in a PCIe evaluation board from Xilinx (KC705) installed in the host PC. The acquired data in the event-based data-path is streamed to the host through the PCIe x8 Direct Memory Access (DMA) with a maximum data throughput per channel is 0.5 GB/s of raw data (event base), 1 GB/s per digitizer and up to 1.6 GB/s in continuous mode. The prototype architecture comprises an host PC with two KC705 modules and four channels, producing up to 2 GB/s in event mode and up to 3.2 GB/s in continuous mode. To reduce the produced data throughput from host to ITER databases, the real-time data compression was evaluated using the LZ4 lossless compression algorithm, which provides compression speed up to 400 MB/s per core. This paper presents the architecture, implementation and test of the parallel real-time data compression system running in multiple isolated cores. The average space saving and the performance results for long term acquisitions up to 30 minutes, using different data block size and different number of CPUs, is also presented.