Entropy, Transinformation and Word Distribution of Information-Carrying Sequences

TL;DR

This paper analyzes the correlations, word distributions, and entropies in information-carrying sequences like texts and music, revealing multi-scale correlations and drawing parallels to nonlinear dynamic systems at critical states.

Contribution

It introduces a detailed analysis of correlations and entropy measures in sequences from texts and music, highlighting their multi-scale structure and analogy to nonlinear systems.

Findings

Sequences exhibit correlations on many length scales.

Word distributions and higher order entropies are characterized.

Analogies to nonlinear systems in critical states are discussed.

Abstract

We investigate correlations in information carriers, e.g. texts and pieces of music, which are represented by strings of letters. For information carrying strings generated by one source (i.e. a novel or a piece of music) we find correlations on many length scales. The word distribution, the higher order entropies and the transinformation are calculated. The analogy to strings generated through symbolic dynamics by nonlinear systems in critical states is discussed.