The LHCb Sprucing and Analysis Productions

TL;DR

This paper details the upgraded offline data processing and analysis system of LHCb for handling increased data volumes, including centralised processing, automation, and tools that improve workflows and analysis preservation, with future plans for HL-LHC.

Contribution

It introduces the LHCb Sprucing and Analysis Productions systems, enhancing offline data handling and automation for large datasets, and discusses future upgrades for HL-LHC.

Findings

Successful implementation of centralised offline processing and streaming.

Automation tools improve analyst workflows and data preservation.

Preparedness for HL-LHC data challenges.

Abstract

The LHCb detector underwent a comprehensive upgrade in preparation for the third data-taking run of the Large Hadron Collider (LHC), known as LHCb Upgrade I. The increased data rate of Run 3 not only posed data collection (Online) challenges but also significant Offline data processing and analysis ones. The offline processing and analysis model was consequently upgraded to handle the factor 30 increase in data volume and the associated demands of ever-growing analyst-level datasets, led by the LHCb Data Processing and Analysis (DPA) project. This paper documents the LHCb "Sprucing" - the centralised offline processing, selections and streaming of data - and "Analysis Productions" - the centralised and highly automated declarative nTuple production system. The DaVinci application used by analysis productions for tupling spruced data is described as well as the apd and lbconda tools for data retrieval and analysis environment configuration. These tools allow for greatly improved analyst workflows and analysis preservation. Finally, the approach to data processing and analysis in the High-Luminosity Large Hadron Collider (HL-LHC) era - LHCb Upgrade II - is discussed.