The impact of lossy data compression on the power spectrum of the high redshift 21-cm signal with LOFAR

TL;DR

This study evaluates lossy data compression using Dysco on LOFAR's high-redshift 21-cm signal data, demonstrating it introduces negligible noise compared to thermal noise, enabling significant data volume reduction without bias.

Contribution

It provides a detailed analysis of Dysco compression effects on the 21-cm power spectrum, establishing optimal settings for minimal impact on scientific results.

Findings

Compression noise is over five orders of magnitude lower than thermal noise.

Compression does not affect calibration or introduce coherent artifacts.

Data volumes can be reduced by a factor of about 4 without bias.

Abstract

Current radio interferometers output multi-petabyte-scale volumes of data per year making the storage, transfer, and processing of this data a sizeable challenge. This challenge is expected to grow with the next-generation telescopes such as the Square Kilometre Array. Lossy compression of interferometric data post-correlation can abate this challenge. However, since high-redshift 21-cm studies impose strict precision requirements, the impact of such lossy data compression on the 21-cm signal power spectrum statistic should be understood. We apply Dysco visibility compression, a technique to normalize and quantize specifically designed for radio interferometric data. We establish the level of the compression noise in the power spectrum in comparison to the thermal noise as well as its coherency behavior. Finally, for optimal compression results, we compare the compression noise obtained from different compression settings to a nominal 21-cm signal power. From a single night of observation, we find that the noise introduced due to the compression is more than five orders of magnitude lower than the thermal noise level in the power spectrum. The noise does not affect calibration. The compression noise shows no correlation with the sky signal and has no measurable coherent component. The level of compression error in the power spectrum ultimately depends on the compression settings. Dysco visibility compression is found to be of insignificant concern for 21-cm power spectrum studies. Hence, data volumes can be safely reduced by factors of $\sim 4$ and with insignificant bias to the final power spectrum. Data from SKA-low will likely be compressible by the same factor as LOFAR, owing to the similarities of the two instruments. The same technique can be used to compress data from other telescopes, but a small adjustment of the compression parameters might be required.