Distributed Data-Processing Pipeline for Mingantu Ultrawide Spectral Radioheliograph

TL;DR

This paper presents a high-performance distributed data processing pipeline for the Chinese Spectral RadioHeliograph, enabling efficient handling of massive observational data for high-resolution solar imaging.

Contribution

It introduces a scalable, robust, and automatic data processing system built on OpenCluster for CSRH, improving data handling and processing efficiency.

Findings

Pipeline meets performance requirements for CSRH data processing

System architecture is comprehensive and adaptable

Supports real-time monitoring and high-resolution imaging

Abstract

The Chinese Spectral RadioHeliograph (CSRH) is a synthetic aperture radio interferometer built in Inner Mongolia, China. As a solar-dedicated interferometric array, CSRH is capable of producing high quality radio images at frequency range from 400 MHz to 15 GHz with high temporal, spatial, and spectral resolution.To implement high cadence imaging at wide-band and obtain more than 2 order higher multiple frequencies, the implementation of the data processing system for CSRH is a great challenge. It is urgent to build a pipeline for processing massive data of CSRH generated every day. In this paper, we develop a high performance distributed data processing pipeline (DDPP) built on the OpenCluster infrastructure for processing CSRH observational data including data storage, archiving, preprocessing, image reconstruction, deconvolution, and real-time monitoring. We comprehensively elaborate the system architecture of the pipeline and the implementation of each subsystem. The DDPP is automatic, robust, scalable and manageable. The processing performance under multi computers parallel and GPU hybrid system meets the requirements of CSRH data processing. The study presents an valuable reference for other radio telescopes especially aperture synthesis telescopes, and also gives an valuable contribution to the current and/or future data intensive astronomical observations.