Undecidability of Underfitting in Learning Algorithms

TL;DR

This paper proves that it is impossible to decide whether a given learning algorithm will always underfit, even with unlimited training, highlighting fundamental limits in understanding algorithm behavior.

Contribution

It establishes the undecidability of predicting underfitting in learning algorithms using an information-theoretic approach, a novel theoretical insight.

Findings

Decidability of underfitting is impossible in general.

Undecidability holds even with unlimited training time.

Highlights limits of algorithm analysis in machine learning.

Abstract

Using recent machine learning results that present an information-theoretic perspective on underfitting and overfitting, we prove that deciding whether an encodable learning algorithm will always underfit a dataset, even if given unlimited training time, is undecidable. We discuss the importance of this result and potential topics for further research, including information-theoretic and probabilistic strategies for bounding learning algorithm fit.