Square Kilometre Array : Processing Voluminous MeerKAT Data on IRIS

TL;DR

This paper discusses the development of data transfer and processing frameworks for handling large MeerKAT astronomical datasets on the UK's IRIS computing infrastructure, enabling efficient analysis outside South Africa.

Contribution

It introduces the RapifXfer data transfer framework and a new processing pipeline tailored for IRIS, facilitating cross-continental data management for MeerKAT.

Findings

Successful transfer of large MeerKAT datasets to IRIS

Implementation of a new processing pipeline for IRIS

Enhanced data management for astronomical research

Abstract

Processing astronomical data often comes with huge challenges with regards to data management as well as data processing. MeerKAT telescope is one of the precursor telescopes of the World's largest observatory - Square Kilometre Array. So far, MeerKAT data was processed using the South African computing facility i.e. IDIA, and exploited to make ground-breaking discoveries. However, to process MeerKAT data on UK's IRIS computing facility requires new implementation of the MeerKAT pipeline. This paper focuses on how to transfer MeerKAT data from the South African site to UK's IRIS systems for processing. We discuss about our RapifXfer Data transfer framework for transferring the MeerKAT data from South Africa to the UK, and the MeerKAT job processing framework pertaining to the UK's IRIS resources.