Developing a Real Time Processing System for HERA

TL;DR

This paper presents the development of a real-time processing system for HERA, a large radio telescope array, enabling efficient data quality inspection, flagging, and initial calibration to handle over 60 TB of nightly data.

Contribution

It introduces a modular, open-source real-time processing system with pipeline management and data handling components tailored for HERA's data volume and analysis needs.

Findings

Successfully processes over 60 TB of data nightly

Provides initial calibration and data quality flagging

Designed for adaptability to other radio astronomy projects

Abstract

The Hydrogen Epoch of Reionization Array (HERA) is a radio telescope in the Karoo desert of South Africa endeavoring to observe Cosmic Dawn and the Epoch of Reionization. When fully constructed, it will consist of 350 antennas and generate over 60 terabytes (TB) of data each night. In order to keep pace with the relatively large rate of data, we have developed the real-time processing (RTP) system for HERA. The RTP is responsible for inspecting data for acceptable levels of quality and flagging unusable data, as well as providing an initial calibration solution. The RTP system consists of the pipeline management system, called the hera_opm package, and the data management system, called the librarian. Though the systems have been developed in the context of analyzing HERA data, they feature public and open source software, and have been designed to be adapted and used in other contexts as necessary.