Stable and Robust Hyper-Parameter Selection Via Robust Information Sharing Cross-Validation

TL;DR

This paper introduces a new cross-validation method for robust linear regression that tracks multiple minima and matches them across data subsets, improving hyper-parameter selection reliability in high-dimensional, outlier-prone settings.

Contribution

It proposes an adaptive CV strategy that enhances hyper-parameter tuning for robust estimators by tracking and matching multiple local minima across data folds.

Findings

Reduces variability of performance estimates

Produces smoother CV curves

Increases reliability of robust penalized estimators

Abstract

Robust estimators for linear regression require non-convex objective functions to shield against adverse affects of outliers. This non-convexity brings challenges, particularly when combined with penalization in high-dimensional settings. Selecting hyper-parameters for the penalty based on a finite sample is a critical task. In practice, cross-validation (CV) is the prevalent strategy with good performance for convex estimators. Applied with robust estimators, however, CV often gives sub-par results due to the interplay between multiple local minima and the penalty. The best local minimum attained on the full training data may not be the minimum with the desired statistical properties. Furthermore, there may be a mismatch between this minimum and the minima attained in the CV folds. This paper introduces a novel adaptive CV strategy that tracks multiple minima for each combination of hyper-parameters and subsets of the data. A matching scheme is presented for correctly evaluating minima computed on the full training data using the best-matching minima from the CV folds. It is shown that the proposed strategy reduces the variability of the estimated performance metric, leads to smoother CV curves, and therefore substantially increases the reliability and utility of robust penalized estimators.