The no-free-lunch theorems of supervised learning

TL;DR

This paper challenges the traditional interpretation of no-free-lunch theorems by emphasizing the importance of model-dependent biases in learning algorithms, suggesting that justified, effective algorithms exist within this framework.

Contribution

It proposes a reinterpretation of no-free-lunch theorems, highlighting the role of model-dependent biases in justifying learning algorithms.

Findings

Standard algorithms are better understood as model-dependent.

Model-relative justification can explain the effectiveness of algorithms.

No-free-lunch theorems do not preclude justified, effective algorithms.

Abstract

The no-free-lunch theorems promote a skeptical conclusion that all possible machine learning algorithms equally lack justification. But how could this leave room for a learning theory, that shows that some algorithms are better than others? Drawing parallels to the philosophy of induction, we point out that the no-free-lunch results presuppose a conception of learning algorithms as purely data-driven. On this conception, every algorithm must have an inherent inductive bias, that wants justification. We argue that many standard learning algorithms should rather be understood as model-dependent: in each application they also require for input a model, representing a bias. Generic algorithms themselves, they can be given a model-relative justification.