On The Sparse Bayesian Learning Of Linear Models

TL;DR

This paper re-examines sparse Bayesian learning for linear regression in high-dimensional settings, proposing a thresholded estimator that achieves optimal error rates and effectively identifies non-zero coefficients, outperforming lasso with strong signals.

Contribution

It introduces a hard-thresholded SBL estimator with proven error bounds and coefficient identification capabilities in high-dimensional linear models.

Findings

Estimator achieves optimal non-asymptotic error rate.

High-probability identification of non-zero coefficients.

Performs better than lasso with strong signals.

Abstract

This work is a re-examination of the sparse Bayesian learning (SBL) of linear regression models of Tipping (2001) in a high-dimensional setting. We propose a hard-thresholded version of the SBL estimator that achieves, for orthogonal design matrices, the non-asymptotic estimation error rate of $\sigma\sqrt{s\log p}/\sqrt{n}$, where $n$ is the sample size, $p$ the number of regressors, $\sigma$ is the regression model standard deviation, and $s$ the number of non-zero regression coefficients. We also establish that with high-probability the estimator identifies the non-zero regression coefficients. In our simulations we found that sparse Bayesian learning regression performs better than lasso (Tibshirani (1996)) when the signal to be recovered is strong.