Minimal penalties and the slope heuristics: a survey

TL;DR

This survey reviews the theoretical foundations, practical implementations, and recent advances of the slope heuristics and minimal penalties in model selection, highlighting their connections, performance, and open problems.

Contribution

It provides a comprehensive overview of slope heuristics, introduces new practical definitions, and discusses their theoretical properties and connections to residual-variance estimators.

Findings

Slope heuristics perform nearly as well as residual-based estimators.

New practical definitions of minimal-penalty algorithms are proposed and tested.

Connections between slope heuristics and classical model selection criteria are established.

Abstract

Birg{\'e} and Massart proposed in 2001 the slope heuristics as a way to choose optimally from data an unknown multiplicative constant in front of a penalty. It is built upon the notion of minimal penalty, and it has been generalized since to some "minimal-penalty algorithms". This paper reviews the theoretical results obtained for such algorithms, with a self-contained proof in the simplest framework, precise proof ideas for further generalizations, and a few new results. Explicit connections are made with residual-variance estimators-with an original contribution on this topic, showing that for this task the slope heuristics performs almost as well as a residual-based estimator with the best model choice-and some classical algorithms such as L-curve or elbow heuristics, Mallows' C p , and Akaike's FPE. Practical issues are also addressed, including two new practical definitions of minimal-penalty algorithms that are compared on synthetic data to previously-proposed definitions. Finally, several conjectures and open problems are suggested as future research directions.