DeepCQ: General-Purpose Deep-Surrogate Framework for Lossy Compression Quality Prediction

TL;DR

DeepCQ is a versatile deep surrogate model that accurately predicts lossy compression quality across various compressors and datasets, reducing computational costs in scientific data management.

Contribution

The paper introduces a generalizable, efficient deep surrogate framework for predicting compression quality, with a novel two-stage design and mixture-of-experts optimization for time-evolving data.

Findings

Prediction errors generally under 10%

Outperforms existing methods significantly

Effective across multiple scientific applications

Abstract

Error-bounded lossy compression techniques have become vital for scientific data management and analytics, given the ever-increasing volume of data generated by modern scientific simulations and instruments. Nevertheless, assessing data quality post-compression remains computationally expensive due to the intensive nature of metric calculations. In this work, we present a general-purpose deep-surrogate framework for lossy compression quality prediction (DeepCQ), with the following key contributions: 1) We develop a surrogate model for compression quality prediction that is generalizable to different error-bounded lossy compressors, quality metrics, and input datasets; 2) We adopt a novel two-stage design that decouples the computationally expensive feature-extraction stage from the light-weight metrics prediction, enabling efficient training and modular inference; 3) We optimize the model performance on time-evolving data using a mixture-of-experts design. Such a design enhances the robustness when predicting across simulation timesteps, especially when the training and test data exhibit significant variation. We validate the effectiveness of DeepCQ on four real-world scientific applications. Our results highlight the framework's exceptional predictive accuracy, with prediction errors generally under 10\% across most settings, significantly outperforming existing methods. Our framework empowers scientific users to make informed decisions about data compression based on their preferred data quality, thereby significantly reducing I/O and computational overhead in scientific data analysis.