A Serverless Engine for High Energy Physics Distributed Analysis

TL;DR

This paper introduces a novel serverless computing engine for High Energy Physics data analysis, leveraging AWS Lambda to enable scalable, distributed processing of large datasets beyond traditional batch systems.

Contribution

It presents a new serverless architecture for HEP data analysis using ROOT and AWS Lambda, addressing scalability challenges of traditional systems.

Findings

Enabled access to CERN datasets through serverless functions

Monitored performance metrics for data- and compute-intensive workloads

Demonstrated scalable distributed analysis outside traditional environments

Abstract

The Large Hadron Collider (LHC) at CERN has generated in the last decade an unprecedented volume of data for the High-Energy Physics (HEP) field. Scientific collaborations interested in analysing such data very often require computing power beyond a single machine. This issue has been tackled traditionally by running analyses in distributed environments using stateful, managed batch computing systems. While this approach has been effective so far, current estimates for future computing needs of the field present large scaling challenges. Such a managed approach may not be the only viable way to tackle them and an interesting alternative could be provided by serverless architectures, to enable an even larger scaling potential. This work describes a novel approach to running real HEP scientific applications through a distributed serverless computing engine. The engine is built upon ROOT, a well-established HEP data analysis software, and distributes its computations to a large pool of concurrent executions on Amazon Web Services Lambda Serverless Platform. Thanks to the developed tool, physicists are able to access datasets stored at CERN (also those that are under restricted access policies) and process it on remote infrastructures outside of their typical environment. The analysis of the serverless functions is monitored at runtime to gather performance metrics, both for data- and computation-intensive workloads.