Self-Regularized Learning Methods

TL;DR

This paper introduces a unified framework called self-regularization that explains implicit complexity control in learning algorithms like gradient descent, providing theoretical analysis and optimal rates without explicit regularization.

Contribution

It presents a general self-regularization framework that encompasses classical and gradient-based algorithms, offering a unified analysis and optimal convergence rates.

Findings

Self-regularization captures implicit complexity control in algorithms.

The framework achieves minmax-optimal statistical rates.

Data-dependent hyperparameter selection methods are analyzed for optimality.

Abstract

We introduce a general framework for analyzing learning algorithms based on the notion of self-regularization, which captures implicit complexity control without requiring explicit regularization. This is motivated by previous observations that many algorithms, such as gradient-descent based learning, exhibit implicit regularization. In a nutshell, for a self-regularized algorithm the complexity of the predictor is inherently controlled by that of the simplest comparator achieving the same empirical risk. This framework is sufficiently rich to cover both classical regularized empirical risk minimization and gradient descent. Building on self-regularization, we provide a thorough statistical analysis of such algorithms including minmax-optimal rates, where it suffices to show that the algorithm is self-regularized -- all further requirements stem from the learning problem itself. Finally, we discuss the problem of data-dependent hyperparameter selection, providing a general result which yields minmax-optimal rates up to a double logarithmic factor and covers data-driven early stopping for RKHS-based gradient descent.