Lossless compression of simulated radio interferometric visibilities

TL;DR

The paper introduces Sisco, a lossless compression method tailored for simulated radio interferometric data, significantly reducing data volume while maintaining data integrity, and compatible with existing workflows.

Contribution

Sisco is a novel lossless compression technique that uses polynomial extrapolation and byte grouping, optimized for noiseless simulated radio interferometric data, and integrates with standard data formats.

Findings

Reduces data volume to 24% on average

Achieves compression ratios from 13% to 38% depending on data complexity

Operates at 534 MB/s throughput, dominated by I/O

Abstract

Context. Processing radio interferometric data often requires storing forward-predicted model data. In direction-dependent calibration, these data may have a volume an order of magnitude larger than the original data. Existing lossy compression techniques work well for observed, noisy data, but cause issues in calibration when applied to forward-predicted model data. Aims. To reduce the volume of forward-predicted model data, we present a lossless compression method called Simulated Signal Compression (Sisco) for noiseless data that integrates seamlessly with existing workflows. We show that Sisco can be combined with baseline-dependent averaging for further size reduction. Methods. Sisco decomposes complex floating-point visibility values and uses polynomial extrapolation in time and frequency to predict values, groups bytes for efficient encoding, and compresses residuals using the Deflate algorithm. We evaluate Sisco on diverse LOFAR, MeerKAT, and MWA datasets with various extrapolation functions. Implemented as an open-source Casacore storage manager, it can directly be used by any observatory that makes use of this format. Results. We find that a combination of linear and quadratic prediction yields optimal compression, reducing noiseless forward-predicted model data to 24% of its original volume on average. Compression varies by dataset, ranging from 13% for smooth data to 38% for less predictable data. For pure noise data, compression achieves just a size of 84% due to the unpredictability of such data. With the current implementation, the achieved compression throughput is with 534 MB/s mostly dominated by I/O on our testing platform, but occupies the processor during compression or decompression. Finally, we discuss the extension to a lossy algorithm.