RawArray: A Simple, Fast, and Extensible Archival Format for Numeric Data

TL;DR

RawArray is a new, simple, and efficient file format designed for storing multidimensional numeric data, offering significant speed advantages over existing formats like HDF5, and supporting extensibility with user metadata.

Contribution

The paper introduces RawArray, a novel archival format that simplifies data storage, improves read speeds, and allows flexible metadata inclusion for scientific and machine learning data.

Findings

2-3x faster than HDF5 in benchmarks

Up to 20x speedup on deep learning datasets

Supports arbitrary user metadata

Abstract

Raw data sizes are growing and proliferating in scientific research, driven by the success of data-hungry computational methods, such as machine learning. The preponderance of proprietary and shoehorned data formats make computations slower and make it harder to reproduce research and to port methods to new platforms. Here we present the RawArray format: a simple, fast, and extensible format for archival storage of multidimensional numeric arrays on disk. The RawArray file format is a simple concatenation of a header array and a data array. The header comprises seven or more 64-bit unsigned integers. The array data can be anything. Arbitrary user metadata can be appended to an RawArray file if desired, for example to store measurement details, color palettes, or geolocation data. We present benchmarks showing a factor of 2--3$\times$ speedup over HDF5 for a range of array sizes and a speedup of up to 20$\times$ in reading the common deep learning datasets MNIST and CIFAR10.