Information Content and Entropy of Finite Patterns from a Combinatorial Perspective

TL;DR

This paper introduces a unified combinatorial framework for defining and comparing the information content and entropy of finite patterns, extending traditional concepts beyond Shannon's limits and applicable to various pattern types.

Contribution

It provides a new combinatorial definition of information and entropy for patterns, compatible with classical concepts and applicable beyond ergodic Markov processes.

Findings

Derived general properties of information content from pattern comparisons

Proposed normalized information estimation methods using compression and Kolmogorov complexity

Redefined entropy from a combinatorial perspective compatible with traditional entropy asymptotically

Abstract

A unified combinatorial definition of the information content and entropy of different types of patterns, compatible with the traditional concepts of information and entropy, going beyond the limitations of Shannon information interpretable for ergodic Markov processes. We compare the information content of various finite patterns and derive general properties of information quantity from these comparisons. Using these properties, we define normalized information estimation methods based on compression algorithms and Kolmogorov complexity. From a combinatorial point of view, we redefine the concept of entropy in a way that is asymptotically compatible with traditional entropy.