A Survey on Error-Bounded Lossy Compression for Scientific Datasets

TL;DR

This survey comprehensively reviews error-bounded lossy compression techniques for scientific datasets, categorizing models, components, and analyzing 46 compressors to guide future research and application development.

Contribution

It introduces a novel taxonomy of compression models, surveys key components, and analyzes state-of-the-art compressors and their application-specific designs.

Findings

Summarizes 6 classic compression models

Analyzes 46 state-of-the-art compressors

Discusses design of application-specific compressors

Abstract

Error-bounded lossy compression has been effective in significantly reducing the data storage/transfer burden while preserving the reconstructed data fidelity very well. Many error-bounded lossy compressors have been developed for a wide range of parallel and distributed use cases for years. They are designed with distinct compression models and principles, such that each of them features particular pros and cons. In this paper we provide a comprehensive survey of emerging error-bounded lossy compression techniques. The key contribution is fourfold. (1) We summarize a novel taxonomy of lossy compression into 6 classic models. (2) We provide a comprehensive survey of 10 commonly used compression components/modules. (3) We summarized pros and cons of 46 state-of-the-art lossy compressors and present how state-of-the-art compressors are designed based on different compression techniques. (4) We discuss how customized compressors are designed for specific scientific applications and use-cases. We believe this survey is useful to multiple communities including scientific applications, high-performance computing, lossy compression, and big data.