Algorithmic information theory

TL;DR

This paper introduces algorithmic information theory, focusing on Kolmogorov complexity, to formalize the distinction between meaningful and random information, and explores its philosophical implications.

Contribution

It provides a comprehensive overview of Kolmogorov complexity and introduces the Kolmogorov structure function to formalize Occam's razor in inductive inference.

Findings

Distinction between structural and random information

Formalization of Occam's razor

Comparison of Kolmogorov and Shannon information theories

Abstract

We introduce algorithmic information theory, also known as the theory of Kolmogorov complexity. We explain the main concepts of this quantitative approach to defining `information'. We discuss the extent to which Kolmogorov's and Shannon's information theory have a common purpose, and where they are fundamentally different. We indicate how recent developments within the theory allow one to formally distinguish between `structural' (meaningful) and `random' information as measured by the Kolmogorov structure function, which leads to a mathematical formalization of Occam's razor in inductive inference. We end by discussing some of the philosophical implications of the theory.