Binary Kernel Logistic Regression: a sparsity-inducing formulation and a convergent decomposition training algorithm

TL;DR

This paper introduces a sparsity-inducing extension of kernel logistic regression with a convergent decomposition training algorithm, balancing prediction accuracy and model sparsity effectively.

Contribution

It proposes a novel sparsity-promoting formulation for binary KLR and a second-order decomposition algorithm with proven global convergence.

Findings

Achieves competitive accuracy and sparsity trade-offs compared to IVM, L1/2 regularization, and SVM.

Provides probabilistic class membership estimates.

Demonstrates effectiveness on 12 benchmark datasets.

Abstract

Kernel logistic regression (KLR) is a widely used supervised learning method for binary and multi-class classification, which provides estimates of the conditional probabilities of class membership for the data points. Unlike other kernel methods such as Support Vector Machines (SVMs), KLRs are generally not sparse. Previous attempts to deal with sparsity in KLR include a heuristic method referred to as the Import Vector Machine (IVM) and ad hoc regularizations such as the $\ell_{1/2}$-based one. Achieving a good trade-off between prediction accuracy and sparsity is still a challenging issue with a potential significant impact from the application point of view. In this work, we revisit binary KLR and propose an extension of the training formulation proposed by Keerthi et al., which is able to induce sparsity in the trained model, while maintaining good testing accuracy. To efficiently solve the dual of this formulation, we devise a decomposition algorithm of Sequential Minimal Optimization type which exploits second-order information, and for which we establish global convergence. Numerical experiments conducted on 12 datasets from the literature show that the proposed binary KLR approach achieves a competitive trade-off between accuracy and sparsity with respect to IVM, $\ell_{1/2}$-based regularization for KLR, and SVM while retaining the advantages of providing informative estimates of the class membership probabilities.