The sky at one terabit per second: Architecture and implementation of the Argus Array Hierarchical Data Processing System

TL;DR

This paper presents the architecture and implementation of the Argus-HDPS, a real-time data processing system for the Argus Optical Array, capable of handling 11 Tbps data rate and producing various data products for astronomical surveys.

Contribution

It introduces a scalable, GPU-enabled hierarchical data processing pipeline for the Argus Array, enabling real-time analysis and transient detection from massive data streams.

Findings

Demonstrated image segmentation and coaddition on representative data.

Achieved real-time transient alerts and deep coadds.

Validated GPU pipelines for high-throughput astronomical data processing.

Abstract

The Argus Optical Array is a synoptic survey observatory, currently in development, that will have a total collecting area equivalent to a 5-meter monolithic telescope and an all-sky field of view, multiplexed from 900 commercial off-the-shelf telescopes. The Array will observe 7916 deg$^2$ every second during high-speed operations ($m_g\leq16.1$) and every 30 seconds at base cadence ($m_g\leq19.1$), producing 4.3 PB and 145 TB respectively of data per night with its 55-gigapixel mosaic of cameras. The Argus Array Hierarchical Data Processing System (Argus-HDPS) is the instrument control and analysis pipeline for the Argus Array project, able to create fully-reduced data products in real time. We pair sub-arrays of cameras with co-located compute nodes, responsible for distilling the raw 11 Tbps data rate into transient alerts, full-resolution image segments around selected targets at 30-second cadence, and full-resolution coadds of the entire field of view at $15+$-min cadences. Production of long-term light curves and transient discovery in deep coadds out to 5-day cadence ($m_g\leq24.0$) will be scheduled for daytime operations. In this paper, we describe the data reduction strategy for the Argus Optical Array and demonstrate image segmentation, coaddition, and difference image analysis using the GPU-enabled Argus-HDPS pipelines on representative data from the Argus Array Technology Demonstrator.