Probabilistic Classification Vector Machine for Multi-Class Classification

TL;DR

This paper introduces a multi-class extension of the probabilistic classification vector machine (PCVM), providing a sparse Bayesian classifier capable of handling multiple classes without heuristic voting strategies.

Contribution

The paper develops the first multi-class PCVM with two learning algorithms, improving multi-class classification performance and probabilistic output consistency.

Findings

Outperforms existing methods on synthetic and benchmark datasets.

Effective in problems with many classes.

Provides probabilistic outputs without contradiction.

Abstract

The probabilistic classification vector machine (PCVM) synthesizes the advantages of both the support vector machine and the relevant vector machine, delivering a sparse Bayesian solution to classification problems. However, the PCVM is currently only applicable to binary cases. Extending the PCVM to multi-class cases via heuristic voting strategies such as one-vs-rest or one-vs-one often results in a dilemma where classifiers make contradictory predictions, and those strategies might lose the benefits of probabilistic outputs. To overcome this problem, we extend the PCVM and propose a multi-class probabilistic classification vector machine (mPCVM). Two learning algorithms, i.e., one top-down algorithm and one bottom-up algorithm, have been implemented in the mPCVM. The top-down algorithm obtains the maximum a posteriori (MAP) point estimates of the parameters based on an expectation-maximization algorithm, and the bottom-up algorithm is an incremental paradigm by maximizing the marginal likelihood. The superior performance of the mPCVMs, especially when the investigated problem has a large number of classes, is extensively evaluated on synthetic and benchmark data sets.