Complexity-Optimized Sparse Bayesian Learning for Scalable Classification Tasks

TL;DR

This paper introduces DQN-SBL, a scalable Sparse Bayesian Learning method that reduces computational complexity from cubic to linear, enabling efficient high-dimensional classification without sacrificing model sparsity or accuracy.

Contribution

The paper proposes DQN-SBL, a novel diagonal Quasi-Newton approach that significantly reduces SBL's computational complexity, making it suitable for large-scale high-dimensional problems.

Findings

DQN-SBL achieves competitive generalization performance.

It maintains high sparsity in the learned models.

The method scales efficiently to large datasets.

Abstract

Sparse Bayesian Learning (SBL) constructs an extremely sparse probabilistic model with very competitive generalization. However, SBL needs to invert a big covariance matrix with complexity $O(M^3)$ (M: feature size) for updating the regularization priors, making it difficult for problems with high dimensional feature space or large data size. As it may easily suffer from the memory overflow issue in such problems. This paper addresses this issue with a newly proposed diagonal Quasi-Newton (DQN) method for SBL called DQN-SBL where the inversion of big covariance matrix is ignored so that the complexity is reduced to $O(M)$. The DQN-SBL is thoroughly evaluated for non linear and linear classifications with various benchmarks of different sizes. Experimental results verify that DQN-SBL receives competitive generalization with a very sparse model and scales well to large-scale problems.