IPComp: Interpolation Based Progressive Lossy Compression for Scientific Applications

TL;DR

IPComp introduces a novel interpolation-based progressive lossy compression method for scientific data, achieving high compression ratios, fast retrieval, and low error, addressing limitations of existing solutions.

Contribution

It is the first to enable interpolation-based algorithms to support progressive retrieval with high efficiency and low cost in scientific data compression.

Findings

Achieves up to 487% higher compression ratios

Faster by up to 698% compared to state-of-the-art methods

Reduces data volume for retrieval by up to 83%

Abstract

Compression is a crucial solution for data reduction in modern scientific applications due to the exponential growth of data from simulations, experiments, and observations. Compression with progressive retrieval capability allows users to access coarse approximations of data quickly and then incrementally refine these approximations to higher fidelity. Existing progressive compression solutions suffer from low reduction ratios or high operation costs, effectively undermining the approach's benefits. In this paper, we propose the first-ever interpolation-based progressive lossy compression solution that has both high reduction ratios and low operation costs. The interpolation-based algorithm has been verified as one of the best for scientific data reduction, but previously no effort exists to make it support progressive retrieval. Our contributions are three-fold: (1) We thoroughly analyze the error characteristics of the interpolation algorithm and propose our solution IPComp with multi-level bitplane and predictive coding. (2) We derive optimized strategies toward minimum data retrieval under different fidelity levels indicated by users through error bounds and bitrates. (3) We evaluate the proposed solution using six real-world datasets from four diverse domains. Experimental results demonstrate our solution archives up to $487\%$ higher compression ratios and $698\%$ faster speed than other state-of-the-art progressive compressors, and reduces the data volume for retrieval by up to $83\%$ compared to baselines under the same error bound, and reduces the error by up to $99\%$ under the same bitrate.