HIDE & SEEK: End-to-End Packages to Simulate and Process Radio Survey Data

TL;DR

This paper introduces two open-source Python packages, HIDE and SEEK, designed to simulate and process radio survey data, enabling better understanding of systematic effects and improving data analysis pipelines for upcoming large-scale radio surveys.

Contribution

The paper presents novel, modular tools for end-to-end simulation and processing of radio survey data, adaptable to different instruments and datasets.

Findings

Successful application to a Galactic survey at 990-1260 MHz

Achieved median SNR of 5-6 in clean channels

Identified challenges with RFI contamination and baseline removal

Abstract

As several large single-dish radio surveys begin operation within the coming decade, a wealth of radio data will become available and provide a new window to the Universe. In order to fully exploit the potential of these data sets, it is important to understand the systematic effects associated with the instrument and the analysis pipeline. A common approach to tackle this is to forward-model the entire system - from the hardware to the analysis of the data products. For this purpose, we introduce two newly developed, open-source Python packages: the HI Data Emulator (HIDE) and the Signal Extraction and Emission Kartographer (SEEK) for simulating and processing single-dish radio survey data. HIDE forward-models the process of collecting astronomical radio signals in a single-dish radio telescope instrument and outputs pixel-level time-ordered-data. SEEK processes the time-ordered-data, removes artifacts from Radio Frequency Interference (RFI), automatically applies flux calibration, and aims to recover the astronomical radio signal. The two packages can be used separately or together depending on the application. Their modular and flexible nature allows easy adaptation to other instruments and data sets. We describe the basic architecture of the two packages and examine in detail the noise and RFI modeling in HIDE, as well as the implementation of gain calibration and RFI mitigation in SEEK. We then apply HIDE & SEEK to forward-model a Galactic survey in the frequency range 990 - 1260 MHz based on data taken at the Bleien Observatory. For this survey, we expect to cover 70% of the full sky and achieve a median signal-to-noise ratio of approximately 5 - 6 in the cleanest channels including systematic uncertainties. However, we also point out the potential challenges of high RFI contamination and baseline removal when examining the early data from the Bleien Observatory.