Empirical entropy in context

TL;DR

This paper explores the historical development and interconnections of empirical entropy with various concepts like Markov processes, Shannon entropy, and Kolmogorov complexity, highlighting its significance in information theory.

Contribution

It provides a comprehensive historical overview of empirical entropy and its relationships with key concepts in information theory and complexity.

Findings

Empirical entropy relates closely to Markov processes and Shannon entropy.

Historical connections between empirical entropy and data compression methods are outlined.

The paper emphasizes the foundational role of empirical entropy in understanding stochastic complexity.

Abstract

We trace the history of empirical entropy, touching briefly on its relation to Markov processes, normal numbers, Shannon entropy, the Chomsky hierarchy, Kolmogorov complexity, Ziv-Lempel compression, de Bruijn sequences and stochastic complexity.