Africanus I. Scalable, distributed and efficient radio data processing with Dask-MS and Codex Africanus

TL;DR

This paper introduces Dask-MS and Codex Africanus, Python libraries that enable scalable, distributed, and efficient radio data processing for large interferometer datasets, balancing performance, flexibility, and ease of development.

Contribution

It presents a novel framework leveraging Dask for distributed radio astronomy data processing, emphasizing open-source tools to handle increasing data volumes and complexity.

Findings

Enables high-performance distributed processing with Dask.

Balances flexibility, performance, and ease-of-use in radio data pipelines.

Facilitates scalable processing on HPC and cloud infrastructures.

Abstract

New radio interferometers such as MeerKAT, SKA, ngVLA, and DSA-2000 drive advancements in software for two key reasons. First, handling the vast data from these instruments requires subdivision and multi-node processing. Second, their improved sensitivity, achieved through better engineering and larger data volumes, demands new techniques to fully exploit it. This creates a critical challenge in radio astronomy software: pipelines must be optimized to process data efficiently, but unforeseen artefacts from increased sensitivity require ongoing development of new techniques. This leads to a trade-off among (1) performance, (2) flexibility, and (3) ease-of-development. Rigid designs often miss the full scope of the problem, while temporary research code is unsuitable for production. This work introduces a framework for developing radio astronomy techniques while balancing the above trade-offs. It prioritizes flexibility and ease-of-development alongside acceptable performance by leveraging Open Source data formats and software. To manage growing data volumes, data is distributed across multiple processors and nodes for parallel processing, utilizing HPC and cloud infrastructure. We present two Python libraries, Dask-MS and Codex Africanus, which enable distributed, high-performance radio astronomy software with Dask. Dask is a lightweight parallelization and distribution framework that integrates with the PyData ecosystem, addressing the "Big Data" challenges of radio astronomy.