A General Framework for Progressive Data Compression and Retrieval

TL;DR

This paper introduces a versatile framework for progressive data compression that enables adaptive, incremental data retrieval with maintained accuracy, regardless of the underlying compression method, improving efficiency in scientific data handling.

Contribution

The authors propose a novel, general framework for progressive data compression that works across different compressors and data representations, supporting adaptive, incremental data retrieval.

Findings

Framework achieves high accuracy compared to standalone compressors.

(De)compression time scales with number of components.

Supports lossless compression with lossy compressors when enough components are used.

Abstract

In scientific simulations, observations, and experiments, the cost of transferring data to and from disk and across networks has become a significant bottleneck that particularly impacts subsequent data analysis and visualization. To address this challenge, compression techniques have been widely adopted. However, traditional lossy compression approaches often require setting error tolerances conservatively to respect the numerical sensitivities of a wide variety of post hoc data analyses, some of which may not even be known a priori. Progressive data compression and retrieval has emerged as a solution, allowing for the adaptive handling of compressed data according to the needs of a given post-processing task. However, few analysis algorithms natively support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. This work presents a general framework that supports progressive-precision data queries independently of the underlying data compressor or number representation. Our approach is based on a multiple-component representation that successively, with each new component, reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented our approach on top of four popular scientific data compressors and have evaluated its behavior on several real-world data sets from the SDRBench collection. Numerical results indicate that our framework is effective in terms of accuracy compared to each of the standalone compressors it builds upon. In addition, (de)compression time is proportional to the number and granularity of components. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.