The Famine of Forte: Few Search Problems Greatly Favor Your Algorithm
George D. Montanez

TL;DR
This paper demonstrates that only a small fraction of problems are favorable for any fixed algorithm, explaining the need for flexible models and new methods in machine learning, supported by bounds based on information theory.
Contribution
It provides theoretical bounds on the proportion of problems favorable to fixed algorithms and introduces a framework linking success probabilities to mutual information and search strategies.
Findings
Favorable problems for a fixed algorithm are strictly limited.
The success probability relates to mutual information between target and data.
Favorable search strategies are also rare and bounded.
Abstract
Casting machine learning as a type of search, we demonstrate that the proportion of problems that are favorable for a fixed algorithm is strictly bounded, such that no single algorithm can perform well over a large fraction of them. Our results explain why we must either continue to develop new learning methods year after year or move towards highly parameterized models that are both flexible and sensitive to their hyperparameters. We further give an upper bound on the expected performance for a search algorithm as a function of the mutual information between the target and the information resource (e.g., training dataset), proving the importance of certain types of dependence for machine learning. Lastly, we show that the expected per-query probability of success for an algorithm is mathematically equivalent to a single-query probability of success under a distribution (called a search…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
