Integrating LHCb workflows on HPC resources: status and strategies

TL;DR

This paper discusses the integration of LHCb workflows into HPC resources, focusing on optimizing performance and distributed computing strategies at the Marconi-A2 system to enhance high-energy physics data processing.

Contribution

It presents novel methods for adapting LHCb workflows to HPC architectures and demonstrates successful multi-process workload submission on a large-scale supercomputer.

Findings

Optimized LHCb workflows for many-core architectures.

Achieved multi-process workload submission in LHCb.

Enhanced HPC utilization for high-energy physics computing.

Abstract

High Performance Computing (HPC) supercomputers are expected to play an increasingly important role in HEP computing in the coming years. While HPC resources are not necessarily the optimal fit for HEP workflows, computing time at HPC centers on an opportunistic basis has already been available to the LHC experiments for some time, and it is also possible that part of the pledged computing resources will be offered as CPU time allocations at HPC centers in the future. The integration of the experiment workflows to make the most efficient use of HPC resources is therefore essential. This paper describes the work that has been necessary to integrate LHCb workflows at a specific HPC site, the Marconi-A2 system at CINECA in Italy, where LHCb benefited from a joint PRACE (Partnership for Advanced Computing in Europe) allocation with the other Large Hadron Collider (LHC) experiments. This has required addressing two types of challenges: on the software application workloads, for optimising their performance on a many-core hardware architecture that differs significantly from those traditionally used in WLCG (Worldwide LHC Computing Grid), by reducing memory footprint using a multi-process approach; and in the distributed computing area, for submitting these workloads using more than one logical processor per job, which had never been done yet in LHCb.