A Bayesian Approach for Online Classifier Ensemble

TL;DR

This paper introduces a Bayesian method for online classifier ensemble learning that recursively updates classifier weights, demonstrating improved convergence and performance over existing methods like stochastic gradient descent and online boosting.

Contribution

It presents a novel Bayesian framework for online classifier weight estimation, offering better convergence rates and empirical performance compared to traditional methods.

Findings

Recursively updates classifier weights using posterior distributions.

Achieves superior convergence rates under stationary data streams.

Outperforms state-of-the-art online learning algorithms in real-world tests.

Abstract

We propose a Bayesian approach for recursively estimating the classifier weights in online learning of a classifier ensemble. In contrast with past methods, such as stochastic gradient descent or online boosting, our approach estimates the weights by recursively updating its posterior distribution. For a specified class of loss functions, we show that it is possible to formulate a suitably defined likelihood function and hence use the posterior distribution as an approximation to the global empirical loss minimizer. If the stream of training data is sampled from a stationary process, we can also show that our approach admits a superior rate of convergence to the expected loss minimizer than is possible with standard stochastic gradient descent. In experiments with real-world datasets, our formulation often performs better than state-of-the-art stochastic gradient descent and online boosting algorithms.