2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization

TL;DR

This paper develops new bounds on the excess risk in empirical risk minimization using local Rademacher complexities, aiding model selection in regression and classification tasks.

Contribution

It introduces general upper bounds on excess risk based on local Rademacher complexities, applicable to various risk minimization frameworks.

Findings

Derived distribution-dependent and data-dependent risk bounds

Bounded excess risk asymptotically in many examples

Applied bounds to model selection in regression and classification

Abstract

Let $\mathcal{F}$ be a class of measurable functions $f:S\mapsto [0,1]$ defined on a probability space $(S,\mathcal{A},P)$. Given a sample (X_1,...,X_n) of i.i.d. random variables taking values in S with common distribution P, let P_n denote the empirical measure based on (X_1,...,X_n). We study an empirical risk minimization problem $P_nf\to \min$, $f\in \mathcal{F}$. Given a solution $\hat{f}_n$ of this problem, the goal is to obtain very general upper bounds on its excess risk \[\mathcal{E}_P(\hat{f}_n):=P\hat{f}_n-\inf_{f\in \mathcal{F}}Pf,\] expressed in terms of relevant geometric parameters of the class $\mathcal{F}$. Using concentration inequalities and other empirical processes tools, we obtain both distribution-dependent and data-dependent upper bounds on the excess risk that are of asymptotically correct order in many examples. The bounds involve localized sup-norms of empirical and Rademacher processes indexed by functions from the class. We use these bounds to develop model selection techniques in abstract risk minimization problems that can be applied to more specialized frameworks of regression and classification.