Data Storage for HEP Experiments in the Era of High-Performance Computing

TL;DR

This paper discusses the challenges of data storage in high-energy physics experiments and explores the potential of HDF5 as an alternative to ROOT for efficient data handling in high-performance computing environments.

Contribution

It provides an overview of the future of HEP data storage, highlighting ongoing efforts to adopt HDF5 for improved scalability and performance in HPC settings.

Findings

HDF5 shows promise for HPC data storage in HEP.

DUNE has adopted HDF5 for data acquisition.

Initial tests indicate potential performance benefits.

Abstract

As particle physics experiments push their limits on both the energy and the intensity frontiers, the amount and complexity of the produced data are also expected to increase accordingly. With such large data volumes, next-generation efforts like the HL-LHC and DUNE will rely even more on both high-throughput (HTC) and high-performance (HPC) computing clusters. Full utilization of HPC resources requires scalable and efficient data-handling and I/O. For the last few decades, ROOT has been used by most HEP experiments to store data. However, other storage technologies like HDF5 may perform better in HPC environments. Initial explorations with HDF5 have begun using ATLAS, CMS and DUNE data; the DUNE experiment has also adopted HDF5 for its data-acquisition system. This paper presents the future outlook of the HEP computing and the role of HPC, and a summary of ongoing and future works to use HDF5 as a possible data storage technology for the HEP experiments to use in HPC environments.