Overfitting Reduction in Convex Regression

TL;DR

This paper addresses overfitting in convex regression by proposing bounded subgradient estimators, providing theoretical guarantees of convergence, and demonstrating improved predictive performance in an electricity distribution application.

Contribution

It introduces two new convex regression estimators with bounded subgradients that reduce overfitting and prove their convergence to the true function.

Findings

Proposed estimators converge uniformly to the true convex function.

Bounded subgradients prevent overfitting near the boundary.

Application shows improved predictive accuracy over existing methods.

Abstract

Convex regression is a method for estimating the convex function from a data set. This method has played an important role in operations research, economics, machine learning, and many other areas. However, it has been empirically observed that convex regression produces inconsistent estimates of convex functions and extremely large subgradients near the boundary as the sample size increases. In this paper, we provide theoretical evidence of this overfitting behavior. To eliminate this behavior, we propose two new estimators by placing a bound on the subgradients of the convex function. We further show that our proposed estimators can reduce overfitting by proving that they converge to the underlying true convex function and that their subgradients converge to the gradient of the underlying function, both uniformly over the domain with probability one as the sample size is increasing to infinity. An application to Finnish electricity distribution firms confirms the superior performance of the proposed methods in predictive power over the existing methods.