Enumerable Distributions, Randomness, Dependence

TL;DR

This paper extends Kolmogorov Complexity to semimeasures to define mutual information for infinite sequences, providing a rigorous theoretical foundation for analyzing randomness and dependence.

Contribution

It introduces a new approach to defining mutual information for infinite sequences using semimeasures, addressing longstanding definitional challenges.

Findings

A tight lower bound for mutual information in terms of Kolmogorov complexity.

Characterization of mutual information via minimization over random sequences.

Framework applicable to theoretical and practical scenarios involving randomness.

Abstract

Mutual information I in infinite sequences (and in their finite prefixes) is essential in theoretical analysis of many situations. Yet its right definition has been elusive for a long time. I address it by generalizing Kolmogorov Complexity theory from measures to SEMImeasures i.e, infimums of sets of measures. Being concave rather than linear functionals, semimeasures are quite delicate to handle. Yet, they adequately grasp various theoretical and practical scenaria. A simple lower bound i$(\alpha:\beta) = \sup\,_{x\in N}\,(K(x) - K(x|\alpha) - K(x|\beta)) $ for information turns out tight for Martin-Lof random $ \alpha,\beta $. For all sequences I$(\alpha:\beta) $ is characterized by the minimum of i$(\alpha':\beta') $ over random $ \alpha',\beta' $ with $ U(\alpha')=\alpha, U(\beta')=\beta $.