Optimal oracle inequalities for model selection

TL;DR

This paper develops general risk bounds for empirical risk minimization in model selection, applicable under weak margin and tail assumptions across regression, classification, and density estimation.

Contribution

It introduces a unified framework for deriving risk bounds under minimal assumptions on margin and tail behavior, extending previous results to broader settings.

Findings

Risk bounds formulated for weak margin conditions

Applicable to regression, classification, and density estimation

General results hold under broad tail assumptions

Abstract

Model selection is often performed by empirical risk minimization. The quality of selection in a given situation can be assessed by risk bounds, which require assumptions both on the margin and the tails of the losses used. Starting with examples from the 3 basic estimation problems, regression, classification and density estimation, we formulate risk bounds for empirical risk minimization under successively weakening conditions and prove them at a very general level, for general margin and power tail behavior of the excess losses.