Application of Lossless Data Compression Techniques to Radio Astronomy Data flows

TL;DR

This paper explores how lossless data compression, based on information theory, can theoretically reduce the cost of transporting and storing large radio astronomy data flows, especially for future instruments like the Square Kilometer Array.

Contribution

It demonstrates that lossless compression of radio astronomy data is theoretically feasible by analyzing data statistics and applying known information theory techniques.

Findings

Lossless compression is possible based on data standard deviation.

Compression can significantly reduce data transport costs.

Potential benefits for storage and data management in large-scale radio astronomy.

Abstract

The modern practice of Radio Astronomy is characterized by extremes of data volume and rates, principally because of the direct relationship between the signal to noise ratio that can be achieved and the need to Nyquist sample the RF bandwidth necessary by way of support. The transport of these data flows is costly. By examining the statistical nature of typical data flows and applying well known techniques from the field of Information Theory the following work shows that lossless compression of typical radio astronomy data flows is in theory possible. The key parameter in determining the degree of compression possible is the standard deviation of the data. The practical application of compression could prove beneficial in reducing the costs of data transport and (arguably) storage for new generation instruments such as the Square Kilometer Array.