Towards Real-World Applications of ServiceX, an Analysis Data Transformation System

TL;DR

This paper discusses ServiceX, a data transformation system designed for high-energy physics experiments, highlighting its new features, integration with analysis frameworks, and readiness for real-world applications in handling large HL-LHC data.

Contribution

The paper introduces new features of ServiceX that enable practical, on-demand data transformation and delivery, facilitating its integration into physics analysis pipelines for HL-LHC data.

Findings

ServiceX can retrieve and transform data on-the-fly from grid sites.

New features make ServiceX ready for public use.

Successful integration with ATLAS analysis frameworks.

Abstract

One of the biggest challenges in the High-Luminosity LHC (HL- LHC) era will be the significantly increased data size to be recorded and analyzed from the collisions at the ATLAS and CMS experiments. ServiceX is a software R&D project in the area of Data Organization, Management and Access of the IRIS- HEP to investigate new computational models for the HL- LHC era. ServiceX is an experiment-agnostic service to enable on-demand data delivery specifically tailored for nearly-interactive vectorized analyses. It is capable of retrieving data from grid sites, on-the-fly data transformation, and delivering user-selected data in a variety of different formats. New features will be presented that make the service ready for public use. An ongoing effort to integrate ServiceX with a popular statistical analysis framework in ATLAS will be described with an emphasis of a practical implementation of ServiceX into the physics analysis pipeline.