The Pivotal Information Criterion

TL;DR

The paper introduces the Pivotal Information Criterion (PIC), a new model selection method that addresses limitations of traditional criteria by using a continuous optimization approach and a data-driven penalty parameter, improving support recovery and model simplicity.

Contribution

The paper proposes PIC, a novel criterion that replaces discrete optimization with continuous, pivotal-based penalty selection, enhancing model selection in high-dimensional settings.

Findings

PIC exhibits a phase transition in support recovery probability.

PIC selects the simplest model among comparable learners.

Simulation results demonstrate improved support detection accuracy.

Abstract

The Bayesian and Akaike information criteria aim at finding a good balance between under- and over-fitting. They are extensively used every day by practitioners. Yet we contend they suffer from at least two afflictions: their penalty parameter $\lambda=\log n$ and $\lambda=2$ are too small, leading to many false discoveries, and their inherent (best subset) discrete optimization is infeasible in high dimension. We alleviate these issues with the pivotal information criterion: PIC is defined as a continuous optimization problem, and the PIC penalty parameter $\lambda$ is selected at the detection boundary (under pure noise). PIC's choice of $\lambda$ is the quantile of a statistic that we prove to be (asymptotically) pivotal, provided the loss function is appropriately transformed. As a result, simulations show a phase transition in the probability of exact support recovery with PIC, a phenomenon studied with no noise in compressed sensing. Applied on real data, for similar predictive performances, PIC selects the least complex model among state-of-the-art learners.