Polycomp: efficient and configurable compression of astronomical timelines

TL;DR

Polycomp is an open-source tool that efficiently compresses astronomical data timelines using polynomial fitting and Chebyshev transforms, achieving high compression ratios with controllable information loss.

Contribution

It introduces a novel, configurable compression algorithm tailored for smooth astronomical data streams, with demonstrated high compression ratios and practical applications.

Findings

Achieves compression ratios up to ~40 for smooth data.

Compresses Ganymede ephemeris with less than 1 meter error.

Reduces Planck LFI data size from 6.5 TB to 0.75 TB.

Abstract

This paper describes the implementation of polycomp, a open-sourced, publicly available program for compressing one-dimensional data series in tabular format. The program is particularly suited for compressing smooth, noiseless streams of data like pointing information, as one of the algorithms it implements applies a combination of least squares polynomial fitting and discrete Chebyshev transforms that is able to achieve a compression ratio Cr up to ~40 in the examples discussed in this work. This performance comes at the expense of a loss of information, whose upper bound is configured by the user. I show two areas in which the usage of polycomp is interesting. In the first example, I compress the ephemeris table of an astronomical object (Ganymede), obtaining Cr ~ 20, with a compression error on the x, y, z coordinates smaller than 1 m. In the second example, I compress the publicly available timelines recorded by the Low Frequency Instrument (LFI), an array of microwave radiometers onboard the ESA Planck spacecraft. The compression reduces the needed storage from ~6.5 TB to ~0.75 TB (C_r ~ 9), thus making them small enough to be kept in a portable hard drive.