High-Dimensional Sparse Bayesian Learning without Covariance Matrices

TL;DR

This paper presents a novel inference method for high-dimensional Sparse Bayesian Learning that avoids large covariance matrices, significantly improving computational efficiency and memory usage in large-scale sparse coding tasks.

Contribution

The authors introduce a new inference scheme combining diagonal estimation and conjugate gradient methods to efficiently perform Sparse Bayesian Learning without explicit covariance matrices.

Findings

Scales better than existing methods in computation time.

Uses less memory, suitable for structured dictionaries.

Maintains accuracy while improving efficiency.

Abstract

Sparse Bayesian learning (SBL) is a powerful framework for tackling the sparse coding problem. However, the most popular inference algorithms for SBL become too expensive for high-dimensional settings, due to the need to store and compute a large covariance matrix. We introduce a new inference scheme that avoids explicit construction of the covariance matrix by solving multiple linear systems in parallel to obtain the posterior moments for SBL. Our approach couples a little-known diagonal estimation result from numerical linear algebra with the conjugate gradient algorithm. On several simulations, our method scales better than existing approaches in computation time and memory, especially for structured dictionaries capable of fast matrix-vector multiplication.