Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data

TL;DR

This paper introduces a hierarchical autoencoder model that effectively compresses large-scale high-resolution scientific data with high ratios and minimal loss, addressing the need for efficient data storage in scientific research.

Contribution

The work presents a novel neural network architecture specifically designed for lossy compression of large-scale scientific data, achieving high compression ratios while maintaining data quality.

Findings

Achieves a compression ratio of 140 on benchmark datasets.

Compresses climate modeling data with a ratio of 200 and negligible error.

Demonstrates effectiveness on high-resolution scientific data.

Abstract

Lossy compression has become an important technique to reduce data size in many domains. This type of compression is especially valuable for large-scale scientific data, whose size ranges up to several petabytes. Although Autoencoder-based models have been successfully leveraged to compress images and videos, such neural networks have not widely gained attention in the scientific data domain. Our work presents a neural network that not only significantly compresses large-scale scientific data, but also maintains high reconstruction quality. The proposed model is tested with scientific benchmark data available publicly and applied to a large-scale high-resolution climate modeling data set. Our model achieves a compression ratio of 140 on several benchmark data sets without compromising the reconstruction quality. 2D simulation data from the High-Resolution Community Earth System Model (CESM) Version 1.3 over 500 years are also being compressed with a compression ratio of 200 while the reconstruction error is negligible for scientific analysis.