A Tiled-Table Convention for Compressing FITS Binary Tables

TL;DR

This paper introduces a new convention for compressing FITS binary tables by subdividing them into tiles and compressing each column within each tile, enhancing data storage efficiency while maintaining easy access.

Contribution

It adapts the FITS tiled-image compression method for binary tables, allowing flexible column-wise compression and preserving original header information.

Findings

Supports efficient data access and storage

Uses gzip for column compression in prototype

Maintains original table structure and headers

Abstract

This document describes a convention for compressing FITS binary tables that is modeled after the FITS tiled-image compression method (White et al. 2009) that has been in use for about a decade. The input table is first optionally subdivided into tiles, each containing an equal number of rows, then every column of data within each tile is compressed and stored as a variable-length array of bytes in the output FITS binary table. All the header keywords from the input table are copied to the header of the output table and remain uncompressed for efficient access. The output compressed table contains the same number and order of columns as in the input uncompressed binary table. There is one row in the output table corresponding to each tile of rows in the input table. In principle, each column of data can be compressed using a different algorithm that is optimized for the type of data within that column, however in the prototype implementation described here, the gzip algorithm is used to compress every column.