Is HDF5 a good format to replace UVFITS?

TL;DR

This paper compares FITS, HDF5, and MeasurementSet formats for radio interferometric data, demonstrating HDF5's advantages and providing a conversion tool, aiming to promote its adoption in radio astronomy.

Contribution

It presents a detailed comparison of data formats and introduces a conversion tool, facilitating the transition to HDF5 for radio astronomy data storage.

Findings

HDF5 offers significant advantages over FITS for data storage.

A conversion tool between FITS, HDF5, and MeasurementSet is developed.

The data model of FITS can be adapted to HDF5.

Abstract

The FITS (Flexible Image Transport System) data format was developed in the late 1970s for storage and exchange of astronomy-related image data. Since then, it has become a standard file format not only for images, but also for radio interferometer data (e.g. UVFITS, FITS-IDI). But is FITS the right format for next-generation telescopes to adopt? The newer Hierarchical Data Format (HDF5) file format offers considerable advantages over FITS, but has yet to gain widespread adoption within radio astronomy. One of the major holdbacks is that HDF5 is not well supported by data reduction software packages. Here, we present a comparison of FITS, HDF5, and the MeasurementSet (MS) format for storage of interferometric data. In addition, we present a tool for converting between formats. We show that the underlying data model of FITS can be ported to HDF5, a first step toward achieving wider HDF5 support.