Aggregation of penalized empirical risk minimizers in regression

TL;DR

This paper investigates the convergence rates of penalized empirical risk minimizers in regression, introduces adaptive estimators, and demonstrates that aggregation improves estimator stability over traditional methods, especially with small samples.

Contribution

It provides a general convergence rate result for PERM, develops adaptive estimators for complex function spaces, and empirically shows aggregation's stability benefits.

Findings

Aggregation yields more stable estimators than cross-validation methods.

Theoretical convergence rates are established for PERM under broad conditions.

Empirical results support aggregation's advantage with small sample sizes.

Abstract

We give a general result concerning the rates of convergence of penalized empirical risk minimizers (PERM) in the regression model. Then, we consider the problem of agnostic learning of the regression, and give in this context an oracle inequality and a lower bound for PERM over a finite class. These results hold for a general multivariate random design, the only assumption being the compactness of the support of its law (allowing discrete distributions for instance). Then, using these results, we construct adaptive estimators. We consider as examples adaptive estimation over anisotropic Besov spaces or reproductive kernel Hilbert spaces. Finally, we provide an empirical evidence that aggregation leads to more stable estimators than more standard cross-validation or generalized cross-validation methods for the selection of the smoothing parameter, when the number of observation is small.