On concentration for (regularized) empirical risk minimization

TL;DR

This paper investigates the concentration behavior of empirical risk minimizers, showing that after normalization, their risk converges to a single point, extending previous results to regularized and more general loss functions.

Contribution

It generalizes and sharpens existing concentration results for empirical risk minimizers to include regularized least squares and other loss functions, using novel proof techniques.

Findings

Risk of empirical minimizer concentrates on a single point after normalization.

Results extend to regularized least squares with convex penalties.

Generalization to other loss functions like negative log-likelihood.

Abstract

Rates of convergence for empirical risk minimizers have been well studied in the literature. In this paper, we aim to provide a complementary set of results, in particular by showing that after normalization, the risk of the empirical minimizer concentrates on a single point. Such results have been established by~\cite{chatterjee2014new} for constrained estimators in the normal sequence model. We first generalize and sharpen this result to regularized least squares with convex penalties, making use of a "direct" argument based on Borell's theorem. We then study generalizations to other loss functions, including the negative log-likelihood for exponential families combined with a strictly convex regularization penalty. The results in this general setting are based on more "indirect" arguments as well as on concentration inequalities for maxima of empirical processes.