A new analytical approach to consistency and overfitting in regularized empirical risk minimization

TL;DR

This paper introduces a novel analytical framework for understanding consistency and overfitting in regularized empirical risk minimization, using tools from modern analysis to rigorously connect overfitting with loss of compactness.

Contribution

It develops a new intrinsic regularized risk functional and provides a concise proof of asymptotic consistency, offering fresh insights into overfitting and underfitting.

Findings

Established asymptotic consistency with regularization parameters shrinking at specific rates

Connected overfitting to loss of compactness in the analytical framework

Provided a new perspective on regularization effects in empirical risk minimization

Abstract

This work considers the problem of binary classification: given training data $x_1, \dots, x_n$ from a certain population, together with associated labels $y_1,\dots, y_n \in \left\{0,1 \right\}$, determine the best label for an element $x$ not among the training data. More specifically, this work considers a variant of the regularized empirical risk functional which is defined intrinsically to the observed data and does not depend on the underlying population. Tools from modern analysis are used to obtain a concise proof of asymptotic consistency as regularization parameters are taken to zero at rates related to the size of the sample. These analytical tools give a new framework for understanding overfitting and underfitting, and rigorously connect the notion of overfitting with a loss of compactness.