Fine Asymptotics for Universal One-to-One Compression of Parametric Sources

TL;DR

This paper introduces a variation of the Type Size code for universal compression of parametric sources, achieving optimal third-order asymptotics by defining type classes based on neighborhoods of minimal sufficient statistics.

Contribution

It proposes a new type class scheme for parametric sources and proves its asymptotic optimality up to the third order, extending previous results to more complex sources.

Findings

The new code outperforms the natural equiprobability-based approach.

Asymptotic analysis confirms the code's optimality up to the third-order rate.

Results apply to both i.i.d. and Markov source models.

Abstract

Universal source coding at short blocklengths is considered for an exponential family of distributions. The \emph{Type Size} code has previously been shown to be optimal up to the third-order rate for universal compression of all memoryless sources over finite alphabets. The Type Size code assigns sequences ordered based on their type class sizes to binary strings ordered lexicographically. To generalize this type class approach for parametric sources, a natural scheme is to define two sequences to be in the same type class if and only if they are equiprobable under any model in the parametric class. This natural approach, however, is shown to be suboptimal. A variation of the Type Size code is introduced, where type classes are defined based on neighborhoods of minimal sufficient statistics. Asymptotics of the overflow rate of this variation are derived and a converse result establishes its optimality up to the third-order term. These results are derived for parametric families of $i.i.d.$ sources as well as Markov sources.