Large Alphabets and Incompressibility

TL;DR

This paper explores the relationship between empirical entropy and Kolmogorov complexity, showing that empirical entropy becomes unreliable for large alphabets when the alphabet size raised to the power of the order exceeds string length.

Contribution

It provides a theoretical analysis of the limitations of empirical entropy as a complexity measure for large alphabets and string lengths.

Findings

Empirical entropy approximates Kolmogorov complexity well for small alphabets.

The approximation breaks down when $n^\ell$ exceeds string length $m$.

The paper offers insights into the applicability of empirical entropy in complexity estimation.

Abstract

We briefly survey some concepts related to empirical entropy -- normal numbers, de Bruijn sequences and Markov processes -- and investigate how well it approximates Kolmogorov complexity. Our results suggest $\ell$th-order empirical entropy stops being a reasonable complexity metric for almost all strings of length $m$ over alphabets of size $n$ about when $n^\ell$ surpasses $m$.