What is important about the No Free Lunch theorems?

TL;DR

The paper discusses the significance of the No Free Lunch theorems in analyzing algorithm performance under non-uniform distributions, highlighting their implications for algorithm comparison, blackbox optimization, and philosophy of science.

Contribution

It clarifies the importance of the theorems in non-uniform scenarios and introduces a translation between supervised learning and blackbox optimization techniques.

Findings

Anti-cross-validation performs similarly to cross-validation without distribution assumptions.

Theorems highlight limitations of algorithm performance claims without distribution assumptions.

A 'dictionary' links supervised learning and blackbox optimization, enabling technique transfer.

Abstract

The No Free Lunch theorems prove that under a uniform distribution over induction problems (search problems or learning problems), all induction algorithms perform equally. As I discuss in this chapter, the importance of the theorems arises by using them to analyze scenarios involving {non-uniform} distributions, and to compare different algorithms, without any assumption about the distribution over problems at all. In particular, the theorems prove that {anti}-cross-validation (choosing among a set of candidate algorithms based on which has {worst} out-of-sample behavior) performs as well as cross-validation, unless one makes an assumption -- which has never been formalized -- about how the distribution over induction problems, on the one hand, is related to the set of algorithms one is choosing among using (anti-)cross validation, on the other. In addition, they establish strong caveats concerning the significance of the many results in the literature which establish the strength of a particular algorithm without assuming a particular distribution. They also motivate a ``dictionary'' between supervised learning and improve blackbox optimization, which allows one to ``translate'' techniques from supervised learning into the domain of blackbox optimization, thereby strengthening blackbox optimization algorithms. In addition to these topics, I also briefly discuss their implications for philosophy of science.