Reimplementing the Hierarchical Data System using HDF5

TL;DR

This paper details the reimplementation of the Starlink Hierarchical Data System (HDS) using HDF5 to enhance accessibility, maintainability, and performance, ensuring long-term sustainability of the data format.

Contribution

The paper introduces a new HDS implementation built on HDF5, addressing legacy limitations and providing a modern, accessible data format for astronomy applications.

Findings

File sizes are comparable between old and new implementations

Performance benchmarks show improvements in data access times

Enhanced compatibility with modern data analysis tools

Abstract

The Starlink Hierarchical Data System has been a very successful niche astronomy file format and library for over 30 years. Development of the library was frozen ten years ago when funding for Starlink was stopped and almost no-one remains who understands the implementation details. To ensure the long-term sustainability of the Starlink application software and to make the extensible N-Dimensional Data Format accessible to a broader range of users, we propose to re-implement the HDS library application interface as a layer on top of the Hierarchical Data Format version 5. We present an overview of the new implementation of version 5 of the HDS file format and describe differences between the expectations of the HDS and HDF5 library interfaces. We finish by comparing the old and new HDS implementations by looking at a comparison of file sizes and by comparing performance benchmarks.