Mitigating multiple descents: A model-agnostic framework for risk monotonization

TL;DR

This paper introduces a model-agnostic framework using cross-validation to modify prediction procedures, ensuring their asymptotic risk decreases monotonically with the aspect ratio, thus mitigating double/multiple descent phenomena.

Contribution

It proposes a general, model-agnostic risk monotonization framework with two data-driven methods, applicable to various prediction procedures and loss functions, without requiring parametric assumptions.

Findings

Framework achieves asymptotic risk monotonicity in high dimensions.

Proposed methods provably mitigate double/multiple descent effects.

Applicable to a broad class of prediction procedures and loss functions.

Abstract

Recent empirical and theoretical analyses of several commonly used prediction procedures reveal a peculiar risk behavior in high dimensions, referred to as double/multiple descent, in which the asymptotic risk is a non-monotonic function of the limiting aspect ratio of the number of features or parameters to the sample size. To mitigate this undesirable behavior, we develop a general framework for risk monotonization based on cross-validation that takes as input a generic prediction procedure and returns a modified procedure whose out-of-sample prediction risk is, asymptotically, monotonic in the limiting aspect ratio. As part of our framework, we propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting, respectively, and show that, under very mild assumptions, they provably achieve monotonic asymptotic risk behavior. Our results are applicable to a broad variety of prediction procedures and loss functions, and do not require a well-specified (parametric) model. We exemplify our framework with concrete analyses of the minimum $\ell_2$, $\ell_1$-norm least squares prediction procedures. As one of the ingredients in our analysis, we also derive novel additive and multiplicative forms of oracle risk inequalities for split cross-validation that are of independent interest.