Adaptive Bayesian Predictive Inference in High-dimensional Regerssion

TL;DR

This paper demonstrates that hierarchical Spike-and-Slab priors enable adaptive, rate-minimax Bayesian predictive inference in high-dimensional sparse regression, outperforming traditional priors like LASSO in predictive tasks.

Contribution

It establishes the first rate-adaptive Bayesian predictive density estimation results for high-dimensional sparse regression using hierarchical Spike-and-Slab priors.

Findings

Hierarchical Spike-and-Slab priors achieve adaptive minimax rates.

LASSO priors are not rate-optimal for predictive distributions.

Bayesian methods outperform traditional priors in predictive accuracy.

Abstract

Bayesian predictive inference provides a coherent description of entire predictive uncertainty through predictive distributions. We examine several widely used sparsity priors from the predictive (as opposed to estimation) inference viewpoint. To start, we investigate predictive distributions in the context of a high-dimensional Gaussian observation with a known variance but an unknown sparse mean under the Kullback-Leibler loss. First, we show that LASSO (Laplace) priors are incapable of achieving rate-optimal predictive distributions. However, deploying the Laplace prior inside the Spike-and-Slab framework (e.g. with the Spike-and-Slab LASSO prior), rate-minimax performance can be attained with properly tuned parameters (depending on the sparsity level sn). We highlight the discrepancy between prior calibration for the purpose of prediction and estimation. Going further, we investigate popular hierarchical priors which are known to attain adaptive rate-minimax performance for estimation. Whether or not they are rate-minimax also for predictive inference has, until now, been unclear. We answer affirmatively by showing that hierarchical Spike-and-Slab priors are adaptive and attain the minimax rate without the knowledge of sn. This is the first rate-adaptive result in the literature on predictive density estimation in sparse setups. Building on the sparse normal-means model, we extend our adaptive rate findings to the case of sparse high-dimensional regression with Spike-and-Slab priors. All of these results underscore benefits of fully Bayesian predictive inference.