Robustness and Generalization

TL;DR

This paper introduces a new approach to understanding the generalization ability of learning algorithms through robustness, establishing it as a fundamental property necessary and sufficient for effective learning.

Contribution

It derives novel generalization bounds based on robustness and demonstrates that robustness is both necessary and sufficient for generalization.

Findings

Robustness is a key property for generalization.

Weak robustness is both necessary and sufficient for learning.

Provides a new perspective beyond complexity or stability arguments.

Abstract

We derive generalization bounds for learning algorithms based on their robustness: the property that if a testing sample is "similar" to a training sample, then the testing error is close to the training error. This provides a novel approach, different from the complexity or stability arguments, to study generalization of learning algorithms. We further show that a weak notion of robustness is both sufficient and necessary for generalizability, which implies that robustness is a fundamental property for learning algorithms to work.