On Probability Estimation by Exponential Smoothing

TL;DR

This paper introduces an adaptive probability estimation method using exponential smoothing for data compression, providing a theoretical analysis that improves redundancy bounds for binary sequences with piecewise stationarity.

Contribution

The paper presents a new exponential smoothing-based probability estimator with a rigorous redundancy analysis, improving bounds for binary data compression models.

Findings

Redundancy is $O(s oot n)$ for $s$ segments, improving over previous $O(s oot{n ext{log} n})$ bounds.

Method runs in constant time per symbol, suitable for real-time applications.

Theoretical analysis applies to binary alphabets with piecewise stationary models.

Abstract

Probability estimation is essential for every statistical data compression algorithm. In practice probability estimation should be adaptive, recent observations should receive a higher weight than older observations. We present a probability estimation method based on exponential smoothing that satisfies this requirement and runs in constant time per letter. Our main contribution is a theoretical analysis in case of a binary alphabet for various smoothing rate sequences: We show that the redundancy w.r.t. a piecewise stationary model with $s$ segments is $O\left(s\sqrt n\right)$ for any bit sequence of length $n$, an improvement over redundancy $O\left(s\sqrt{n\log n}\right)$ of previous approaches with similar time complexity.