Scalable Multi-Class Bayesian Support Vector Machines for Structured and Unstructured Data

TL;DR

This paper presents a scalable Bayesian multi-class support vector machine that effectively handles both structured and unstructured data, providing accurate classification and uncertainty estimation.

Contribution

It introduces a novel Bayesian multi-class SVM with a variational inference approach and an inducing point approximation for large datasets, plus hybrid neural network models for unstructured data.

Findings

Outperforms competitors in training time and accuracy on multiple datasets

Provides reliable uncertainty estimates for active learning and adversarial detection

Scales efficiently to large datasets with the proposed approximation methods

Abstract

We introduce a new Bayesian multi-class support vector machine by formulating a pseudo-likelihood for a multi-class hinge loss in the form of a location-scale mixture of Gaussians. We derive a variational-inference-based training objective for gradient-based learning. Additionally, we employ an inducing point approximation which scales inference to large data sets. Furthermore, we develop hybrid Bayesian neural networks that combine standard deep learning components with the proposed model to enable learning for unstructured data. We provide empirical evidence that our model outperforms the competitor methods with respect to both training time and accuracy in classification experiments on 68 structured and two unstructured data sets. Finally, we highlight the key capability of our model in yielding prediction uncertainty for classification by demonstrating its effectiveness in the tasks of large-scale active learning and detection of adversarial images.