Automatic data processing for Baikal-GVD neutrino observatory

TL;DR

The paper describes the development of an automated data processing pipeline for the Baikal-GVD neutrino observatory, enabling efficient handling of large data volumes through modular, parallelized software components.

Contribution

It introduces the Baikal Analysis and Reconstruction software (BARS), a comprehensive, modular system for automatic data processing in a large-scale neutrino observatory.

Findings

Processing several hours after run completion

Parallel execution of resource-intensive programs

Centralized database for metadata and status monitoring

Abstract

Baikal-GVD is a gigaton-scale neutrino observatory under construction in Lake Baikal. It currently produces about 100 GB of data every day. For their automatic processing, the Baikal Analysis and Reconstruction software (BARS) was developed. At the moment, it includes such stages as hit extraction from PMT waveforms, assembling events from raw data, assigning timestamps to events, determining the position of the optical modules using an acoustic positioning system, data quality monitoring, muon track and cascade reconstruction, as well as the alert signal generation. These stages are implemented as C++ programs which are executed sequentially one after another and can be represented as a directed acyclic graph. The most resource-consuming programs run in parallel to speed up processing. A separate Python package based on the luigi package is responsible for program execution control. Additional information such as the program execution status and run metadata are saved into a central database and then displayed on the dashboard. Results can be obtained several hours after the run completion.