HEP Software Foundation Community White Paper Working Group - Data Processing Frameworks

TL;DR

This white paper discusses the importance of data processing frameworks in high-energy physics experiments, highlighting challenges posed by modern parallel and heterogeneous computing environments, and proposing a research agenda to address these issues.

Contribution

It identifies key concepts and challenges of HEP data frameworks and outlines a program of research and development to improve their scalability and maintainability.

Findings

Frameworks are crucial for physics data processing in HEP.

Modern computing landscapes require frameworks to adapt to parallelism and heterogeneity.

A proposed R&D agenda aims to enhance framework capabilities.

Abstract

Data processing frameworks are an essential part of HEP experiments' software stacks. Frameworks provide a means by which code developers can undertake the essential tasks of physics data processing, accessing relevant inputs and storing their outputs, in a coherent way without needing to know the details of other domains. Frameworks provide essential core services for developers and help deliver a configurable working application to the experiments' production systems. Modern HEP processing frameworks are in the process of adapting to a new computing landscape dominated by parallel processing and heterogeneity, which pose many questions regarding enhanced functionality and scaling that must be faced without compromising the maintainability of the code. In this paper we identify a program of work that can help further clarify the key concepts of frameworks for HEP and then spawn R&D activities that can focus the community's efforts in the most efficient manner to address the challenges of the upcoming experimental program.