Patterns of i.i.d. Sequences and Their Entropy - Part II: Bounds for Some Distributions

TL;DR

This paper provides precise entropy bounds and approximations for patterns of i.i.d. sequences generated by uniform, monotonic, and geometric distributions, including cases with infinite entropy rates.

Contribution

It introduces accurate approximations and bounds for pattern block entropies of i.i.d. sequences from specific distributions, extending previous general bounds.

Findings

Derived precise entropy approximations for uniform and monotonic distributions

Provided numerical bounds for short sequence blocks

Achieved tight bounds even for distributions with infinite entropy rates

Abstract

A pattern of a sequence is a sequence of integer indices with each index describing the order of first occurrence of the respective symbol in the original sequence. In a recent paper, tight general bounds on the block entropy of patterns of sequences generated by independent and identically distributed (i.i.d.) sources were derived. In this paper, precise approximations are provided for the pattern block entropies for patterns of sequences generated by i.i.d. uniform and monotonic distributions, including distributions over the integers, and the geometric distribution. Numerical bounds on the pattern block entropies of these distributions are provided even for very short blocks. Tight bounds are obtained even for distributions that have infinite i.i.d. entropy rates. The approximations are obtained using general bounds and their derivation techniques. Conditional index entropy is also studied for distributions over smaller alphabets.