Synchronizing Probabilities in Model-Driven Lossless Compression

TL;DR

This paper introduces PMATIC, a novel probabilistic coding method that tolerates prediction mismatches in model-driven lossless compression, ensuring robustness and improved compression rates with theoretical guarantees and empirical validation.

Contribution

The paper formalizes prediction mismatch issues in model-driven compression and proposes PMATIC, a model-agnostic algorithm that maintains correctness and efficiency despite bounded prediction errors.

Findings

PMATIC is robust to prediction mismatch in compression.

PMATIC outperforms standard compression tools on text data.

Theoretical bounds confirm PMATIC's effectiveness.

Abstract

It is well-known in the field of lossless data compression that probabilistic next-symbol prediction can be used to compress sequences of symbols. Deep neural networks are able to capture rich dependencies in data, offering a powerful means of estimating these probabilities and hence an avenue towards more effective compression algorithms. However, both compressor and decompressor must have exactly matching predictions; even small differences from non-determinism (which often happen with learned models due to hardware, software, or computation order) can lead to cascading decoding failures. In this paper, we formalize the problem of prediction mismatch in model-driven compression, and introduce Probability Matching Interval Coding (PMATIC), a model-agnostic algorithm that tolerates bounded prediction mismatch with low overhead. PMATIC works with the predicted probabilities, making it compatible as a drop-in replacement for the arithmetic encoder in model-driven compression tools. We show theoretical correctness and performance bounds for PMATIC, and validate these results on text data. These results confirm that, when paired an advanced prediction model, PMATIC is robust to prediction mismatch while achieving compression rates that out-perform standard modern compression tools.