Fast and Exact: Asymptotically Linear KL-Optimal Frequency Normalization

TL;DR

This paper introduces three provably KL-optimal algorithms for frequency normalization in range coders and ANS, achieving asymptotic linearity and improving upon heuristic methods.

Contribution

The paper presents three new algorithms that are provably KL-optimal and asymptotically linear, surpassing existing heuristic and marginally optimal normalizers.

Findings

Three algorithms are proven KL-optimal.

One algorithm runs in O(r), asymptotically optimal.

Algorithms outperform heuristic normalizers.

Abstract

Range coders and ANS replace empirical probabilities with integer frequencies summing to a fixed $M$; the resulting per-symbol code-length redundancy is exactly the KL divergence of the empirical distribution from the quantized one. Existing normalizers (Giesen, Bloom, Collet) are heuristic or only partially marginal-optimal. We give three provably KL-optimal algorithms: a bottom-up archetype, a bidirectional exchange repair of Bloom's heap correction, and a top-down window method that runs in $\mathcal{O}(r)$, asymptotically optimal in $r$, where $r$ is the number of positive-count symbols.