Sparse Multinomial Logistic Regression via Approximate Message Passing

TL;DR

This paper introduces approximate message passing algorithms for sparse multinomial logistic regression, improving classification accuracy and computational efficiency in multi-class problems with feature selection.

Contribution

It develops novel HyGAMP-based algorithms for sparse MLR, including simplified variants and hyperparameter tuning methods, advancing state-of-the-art performance.

Findings

Improved error-rate on synthetic and real datasets.

Reduced runtime compared to existing methods.

Effective feature selection in multi-class classification.

Abstract

For the problem of multi-class linear classification and feature selection, we propose approximate message passing approaches to sparse multinomial logistic regression (MLR). First, we propose two algorithms based on the Hybrid Generalized Approximate Message Passing (HyGAMP) framework: one finds the maximum a posteriori (MAP) linear classifier and the other finds an approximation of the test-error-rate minimizing linear classifier. Then we design computationally simplified variants of these two algorithms. Next, we detail methods to tune the hyperparameters of their assumed statistical models using Stein's unbiased risk estimate (SURE) and expectation-maximization (EM), respectively. Finally, using both synthetic and real-world datasets, we demonstrate improved error-rate and runtime performance relative to existing state-of-the-art approaches to sparse MLR.