No Free Lunch versus Occam's Razor in Supervised Learning

TL;DR

This paper argues for a universal bias in supervised learning algorithms using algorithmic information theory, introduces a new off-line classification algorithm inspired by Solomonoff induction, and supports the heuristic of random training data selection.

Contribution

It presents a theoretical argument for a universal bias in algorithms and introduces a new classification method based on Solomonoff induction principles.

Findings

Universal bias enables success across diverse problem domains.

The new algorithm performs well on structured problems under certain assumptions.

Randomly selecting training data can effectively reduce misclassification rates.

Abstract

The No Free Lunch theorems are often used to argue that domain specific knowledge is required to design successful algorithms. We use algorithmic information theory to argue the case for a universal bias allowing an algorithm to succeed in all interesting problem domains. Additionally, we give a new algorithm for off-line classification, inspired by Solomonoff induction, with good performance on all structured problems under reasonable assumptions. This includes a proof of the efficacy of the well-known heuristic of randomly selecting training data in the hope of reducing misclassification rates.