High-Performance Data Format for Scientific Data Storage and Analysis

TL;DR

This paper introduces the HiPO data format, a high-performance, versatile storage solution for large-scale nuclear physics experimental data, optimized for efficiency and multi-language accessibility.

Contribution

The paper presents the design, implementation, and performance comparison of the HiPO data format, a novel solution for efficient data storage and analysis in nuclear physics experiments.

Findings

HiPO offers improved data compression and access speeds.

It supports multiple programming languages and analysis frameworks.

Performance benchmarks show advantages over ROOT and Parquet formats.

Abstract

In this article, we present the High-Performance Output (HiPO) data format developed at Jefferson Laboratory for storing and analyzing data from Nuclear Physics experiments. The format was designed to efficiently store large amounts of experimental data, utilizing modern fast compression algorithms. The purpose of this development was to provide organized data in the output, facilitating access to relevant information within the large data files. The HiPO data format has features that are suited for storing raw detector data, reconstruction data, and the final physics analysis data efficiently, eliminating the need to do data conversions through the lifecycle of experimental data. The HiPO data format is implemented in C++ and JAVA, and provides bindings to FORTRAN, Python, and Julia, providing users with the choice of data analysis frameworks to use. In this paper, we will present the general design and functionalities of the HiPO library and compare the performance of the library with more established data formats used in data analysis in High Energy and Nuclear Physics (such as ROOT and Parquete).