Which Algorithms Have Tight Generalization Bounds?

TL;DR

This paper investigates conditions under which machine learning algorithms have tight generalization bounds, highlighting the roles of stability and inductive biases in determining these bounds.

Contribution

It provides a characterization linking tight generalization bounds to the stability and conditional variance of algorithms' loss functions.

Findings

Algorithms with certain unstable inductive biases lack tight bounds.

Stable algorithms do have tight generalization bounds.

Tight bounds relate to the conditional variance of the loss.

Abstract

We study which machine learning algorithms have tight generalization bounds. First, we present conditions that preclude the existence of tight generalization bounds. Specifically, we show that algorithms that have certain inductive biases that cause them to be unstable do not admit tight generalization bounds. Next, we show that algorithms that are sufficiently stable do have tight generalization bounds. We conclude with a simple characterization that relates the existence of tight generalization bounds to the conditional variance of the algorithm's loss.