Deep Investigation of Neutral Gas Origins (DINGO): Options for the Processing and Storage of Radio Astronomy Data for robust Deep Spectral Line Imaging in the SKA-Era using uv-Grids

TL;DR

This paper presents a new data processing approach for radio astronomy that uses uv-grid data compression to address storage and I/O challenges in large spectral line surveys, enabling faster processing and maintaining data quality.

Contribution

It introduces a pipeline that employs lossy and lossless compression of uv-grid data, significantly improving processing speed and storage efficiency for large-scale radio astronomy datasets.

Findings

Data compression ratios up to 20 with lossy methods.

Processing speed increased by a factor of 7.

Effective compression within error bounds preserves data quality.

Abstract

The next generation of radio astronomy telescopes are challenging existing data analysis paradigms, as they have an order of magnitude more antennas and larger bandwidth. Foremost amongst these are deep spectral line surveys, because these have the largest number of epochs and spectral channels per dataset. For example, the Deep Investigation of Neutral Gas Origins (DINGO) project on the Australian Square Kilometre Array Pathfinder (ASKAP) aims to observe over 3,000 hours spread over hundreds of observing sessions, covering two pointings and two frequency settings. The two primary problems encountered when processing this data are the need for storage and that processing is primarily I/O limited. To address these issues, we have implemented an deep imaging pipeline based on the storage of an intermediate data product in the software ASKAPSoft, that of the uv-gridded data, and have demonstrated lossy and lossless compression of this data on ASKAP, using MGARD and ADIOS2 libraries. We find data compression ratios from a factor of 7 (lossless) up to 20 (using lossy compression with an absolute error bound of $10^{-4}$), and processing is faster by a factor of 7 for lossless compression. We discuss the effectiveness of lossy MGARD compression and its adherence to the designated error bounds, the trade-off between these error bounds and the corresponding compression ratios, as well as the potential consequences of these I/O and storage improvements on the science quality of the data products.