NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression

TL;DR

NeurLZ introduces an online neural learning-based approach that adaptively enhances scientific lossy compression by integrating lightweight models, error regulation, and cross-field learning, significantly improving compression efficiency and data quality.

Contribution

This work presents NeurLZ, a novel online neural compression method that adapts in real-time, leveraging cross-field correlations and error regulation to outperform existing compressors.

Findings

Achieves up to 94% bit rate reduction at equivalent distortion.

Outperforms state-of-the-art compressors on HPC datasets.

Effectively recovers fine details lost in traditional compression.

Abstract

Large-scale scientific simulations generate massive datasets, posing challenges for storage and I/O. Traditional lossy compression struggles to advance more in balancing compression ratio, data quality, and adaptability to diverse scientific data features. While deep learning-based solutions have been explored, their common practice of relying on large models and offline training limits adaptability to dynamic data characteristics and computational efficiency. To address these challenges, we propose NeurLZ, a neural method designed to enhance lossy compression by integrating online learning, cross-field learning, and robust error regulation. Key innovations of NeurLZ include: (1) compression-time online neural learning with lightweight skipping DNN models, adapting to residual errors without costly offline pertaining, (2) the error-mitigating capability, recovering fine details from compression errors overlooked by conventional compressors, (3) $1\times$ and $2\times$ error-regulation modes, ensuring strict adherence to $1\times$ user-input error bounds strictly or relaxed 2$\times$ bounds for better overall quality, and (4) cross-field learning leveraging inter-field correlations in scientific data to improve conventional methods. Comprehensive evaluations on representative HPC datasets, e.g., Nyx, Miranda, Hurricane, against state-of-the-art compressors show NeurLZ's effectiveness. During the first five learning epochs, NeurLZ achieves an 89% bit rate reduction, with further optimization yielding up to around 94% reduction at equivalent distortion, significantly outperforming existing methods, demonstrating NeurLZ's superior performance in enhancing scientific lossy compression as a scalable and efficient solution.