Algorithmic stability and hypothesis complexity

TL;DR

This paper introduces a new concept called argument stability to analyze how the stability of learning algorithms affects their ability to generalize, providing bounds applicable to algorithms like empirical risk minimization and stochastic gradient descent.

Contribution

It proposes a novel stability notion in Banach spaces and derives generalization bounds based on martingale inequalities, extending stability analysis to a broader class of algorithms.

Findings

Argument stability bounds generalization error in learning algorithms.

Martingale inequalities in Banach spaces underpin the bounds.

Applicable to empirical risk minimization and stochastic gradient descent.

Abstract

We introduce a notion of algorithmic stability of learning algorithms---that we term \emph{argument stability}---that captures stability of the hypothesis output by the learning algorithm in the normed space of functions from which hypotheses are selected. The main result of the paper bounds the generalization error of any learning algorithm in terms of its argument stability. The bounds are based on martingale inequalities in the Banach space to which the hypotheses belong. We apply the general bounds to bound the performance of some learning algorithms based on empirical risk minimization and stochastic gradient descent.