Enhancing ZFP: A Statistical Approach to Understanding and Reducing Error Bias in a Lossy Floating-Point Compression Algorithm

TL;DR

This paper analyzes the error bias in ZFP lossy compression for floating-point data, revealing bias issues and proposing modifications to neutralize bias and improve accuracy in scientific data compression.

Contribution

It provides a statistical analysis of ZFP compression error and introduces modifications to reduce bias and improve compression accuracy.

Findings

Error in ZFP compression is biased.

Proposed modifications neutralize error bias.

Reduced error improves data fidelity.

Abstract

The amount of data generated and gathered in scientific simulations and data collection applications is continuously growing, putting mounting pressure on storage and bandwidth concerns. A means of reducing such issues is data compression; however, lossless data compression is typically ineffective when applied to floating-point data. Thus, users tend to apply a lossy data compressor, which allows for small deviations from the original data. It is essential to understand how the error from lossy compression impacts the accuracy of the data analytics. Thus, we must analyze not only the compression properties but the error as well. In this paper, we provide a statistical analysis of the error caused by ZFP compression, a state-of-the-art, lossy compression algorithm explicitly designed for floating-point data. We show that the error is indeed biased and propose simple modifications to the algorithm to neutralize the bias and further reduce the resulting error.