Optimising NGAS for the MWA Archive

TL;DR

This paper discusses the optimization of NGAS, an open-source archive system, to handle the high data throughput and management challenges of the MWA radio telescope's large-scale data archive.

Contribution

The paper presents tailored optimizations of NGAS for the MWA, enabling high-throughput data ingestion and efficient multi-tiered data management.

Findings

Achieved high data ingestion rates for MWA data.

Enhanced dataflow management across distributed storage tiers.

Improved data staging for processing efficiency.

Abstract

The Murchison Widefield Array (MWA) is a next-generation radio telescope, generating visibility data products continuously at about 400 MB/s. Efficiently managing and archiving this data is a challenge. The MWA Archive consists of dataflows and storage sub-systems distributed across three tiers. At its core is the open source software - the Next-Generation Archive System (NGAS) - that was initially developed in ESO. However, to meet the MWA data challenge, we have tailored and optimised NGAS to achieve high-throughput data ingestion, efficient dataflow management, multi-tiered data storage and processing-aware data staging.