Predictive Criteria for Prior Selection Using Shrinkage in Linear Models

TL;DR

This paper introduces a predictive stability criterion for selecting shrinkage penalties in linear models, offering a data-driven approach that adapts to finite samples and can construct priors, with demonstrated effectiveness through simulations and real data.

Contribution

It proposes a novel predictive stability criterion for penalty and prior selection, including a genetic algorithm-based method for customizing penalties in linear models.

Findings

Custom penalties generally outperform standard ones.

The method constructs priors that are never worse than common choices.

Predictive stability is effective in finite sample settings.

Abstract

Choosing a shrinkage method can be done by selecting a penalty from a list of pre-specified penalties or by constructing a penalty based on the data. If a list of penalties for a class of linear models is given, we provide comparisons based on sample size and number of non-zero parameters under a predictive stability criterion based on data perturbation. These comparisons provide recommendations for penalty selection in a variety of settings. If the preference is to construct a penalty customized for a given problem, then we propose a technique based on genetic algorithms, again using a predictive criterion. We find that, in general, a custom penalty never performs worse than any commonly used penalties but that there are cases the custom penalty reduces to a recognizable penalty. Since penalty selection is mathematically equivalent to prior selection, our method also constructs priors. The techniques and recommendations we offer are intended for finite sample cases. In this context, we argue that predictive stability under perturbation is one of the few relevant properties that can be invoked when the true model is not known. Nevertheless, we study variable inclusion in simulations and, as part of our shrinkage selection strategy, we include oracle property considerations. In particular, we see that the oracle property typically holds for penalties that satisfy basic regularity conditions and therefore is not restrictive enough to play a direct role in penalty selection. In addition, our real data example also includes considerations merging from model mis-specification.