Error-controlled Progressive Retrieval of Scientific Data under Derivable Quantities of Interest

TL;DR

This paper introduces a novel progressive data retrieval framework that guarantees error control on derived quantities of interest, improving data transfer efficiency while maintaining specified accuracy bounds.

Contribution

It provides a generic theory for controlling QoI errors during progressive retrieval and develops an optimized framework applicable to various QoIs.

Findings

Achieves over 2.02x data transfer performance gain.

Faithfully respects user-specified QoI error bounds.

Demonstrates effectiveness on five real-world datasets.

Abstract

The unprecedented amount of scientific data has introduced heavy pressure on the current data storage and transmission systems. Progressive compression has been proposed to mitigate this problem, which offers data access with on-demand precision. However, existing approaches only consider precision control on primary data, leaving uncertainties on the quantities of interest (QoIs) derived from it. In this work, we present a progressive data retrieval framework with guaranteed error control on derivable QoIs. Our contributions are three-fold. (1) We carefully derive the theories to strictly control QoI errors during progressive retrieval. Our theory is generic and can be applied to any QoIs that can be composited by the basis of derivable QoIs proved in the paper. (2) We design and develop a generic progressive retrieval framework based on the proposed theories, and optimize it by exploring feasible progressive representations. (3) We evaluate our framework using five real-world datasets with a diverse set of QoIs. Experiments demonstrate that our framework can faithfully respect any user-specified QoI error bounds in the evaluated applications. This leads to over 2.02x performance gain in data transfer tasks compared to transferring the primary data while guaranteeing a QoI error that is less than 1E-5.