Sequential Universal Modeling for Non-Binary Sequences with Constrained Distributions

TL;DR

This paper introduces a sequential probability assignment algorithm for non-binary sequences with constrained distributions, achieving asymptotic optimality in universal compression for large alphabets.

Contribution

It extends universal compression techniques to large alphabets with bounded parameters, providing a modified KT estimator with proven asymptotic optimality.

Findings

Asymptotic optimality of the proposed algorithm up to O(1) for maximal and average redundancies.

Precise analysis of minimax redundancies for constrained distributions.

Extension of binary case results to larger alphabets.

Abstract

Sequential probability assignment and universal compression go hand in hand. We propose sequential probability assignment for non-binary (and large alphabet) sequences with empirical distributions whose parameters are known to be bounded within a limited interval. Sequential probability assignment algorithms are essential in many applications that require fast and accurate estimation of the maximizing sequence probability. These applications include learning, regression, channel estimation and decoding, prediction, and universal compression. On the other hand, constrained distributions introduce interesting theoretical twists that must be overcome in order to present efficient sequential algorithms. Here, we focus on universal compression for memoryless sources, and present precise analysis for the maximal minimax and the average minimax for constrained distributions. We show that our sequential algorithm based on modified Krichevsky-Trofimov (KT) estimator is asymptotically optimal up to $O(1)$ for both maximal and average redundancies. This paper follows and addresses the challenge presented in \cite{stw08} that suggested "results for the binary case lay the foundation to studying larger alphabets".