The Real-Time Data Processor Framework for Data Handling and Analysis of High-Energy Instruments

TL;DR

This paper introduces a flexible, real-time data processing framework based on ZeroMQ, designed for high-throughput data streams in astrophysics and high-energy instrument applications, with built-in monitoring and configurability.

Contribution

The paper presents a novel, adaptable real-time data processor framework that supports distributed analysis and monitoring for high-energy astrophysical instruments.

Findings

Successfully used in multiple astrophysical projects

Supports high-throughput data handling and analysis

Provides comprehensive monitoring and configuration features

Abstract

We implemented a real-time data processor (rta-dp) framework that can be used to develop real-time analysis pipelines and data handling systems to manage high-throughput data streams with distributed applications in the context of ground and space astrophysical projects and high-energy instruments. The rta-dp is based on the ZeroMQ in-memory communication framework to receive input data, share data between distributed processes, and send or receive commands and pipeline configuration. The rta-dp framework has a flexible architecture that allows the implementation of distributed analysis systems customized to the requirements of several scenarios. The rta-dp framework also provides monitoring capabilities for the running processes and sends housekeeping, logging, alarms, and informative messages that a monitoring process can acquire. We are using the rta-dp in several contexts, such as acquiring and processing data from X-ray detectors to the data quality system of the ASTRI Project, as well as reprocessing and archiving data.