Complexity Regularization and Local Metric Entropy

TL;DR

This paper develops a unified theoretical framework for structural risk minimization in regression, introducing complexity penalties and new consistency results for selecting estimators from function classes.

Contribution

It generalizes the consistency of structural risk minimization in regression and proposes a criterion based on empirical error plus complexity penalty for estimator selection.

Findings

Proposes a criterion for estimator selection using complexity penalties.

Provides new theoretical results on the consistency of structural risk minimization.

Unifies existing literature within a comprehensive regression estimation framework.

Abstract

In the context of Structural Risk Minimization, one is presented a sequence of classes $\{\mathcal{G}_j\}$ from which, given a random sample $(X_i,Y_i)$ one wants to choose a strongly consistent estimator. For certain types of classes of functions, we present a criterion to choose an estimator, based on the minimization of the sum of empirical error and a complexity penalty $r(n,j)$ over each class $\G_j$. We present also several other results together with important results found on current literature on the subject, in an attempt to present and unify the theory in the context of regression estimation. In particular we present a generalization of the consistency of Structural Risk Minimization in the context of regression estimation which are all new results found on Chapter 5.