Comparison of the Bayesian and Randomised Decision Tree Ensembles within an Uncertainty Envelope Technique

TL;DR

This paper compares Bayesian and randomized decision tree ensembles in terms of classification uncertainty, demonstrating Bayesian methods' superiority within an uncertainty envelope framework on various datasets.

Contribution

It provides an experimental comparison of Bayesian model averaging and randomized decision tree ensembles using an uncertainty envelope technique, highlighting the advantages of Bayesian approaches.

Findings

Bayesian decision trees outperform randomized ensembles in uncertainty estimation.

The uncertainty envelope technique effectively evaluates classification confidence.

Bayesian methods provide more realistic uncertainty estimates.

Abstract

Multiple Classifier Systems (MCSs) allow evaluation of the uncertainty of classification outcomes that is of crucial importance for safety critical applications. The uncertainty of classification is determined by a trade-off between the amount of data available for training, the classifier diversity and the required performance. The interpretability of MCSs can also give useful information for experts responsible for making reliable classifications. For this reason Decision Trees (DTs) seem to be attractive classification models for experts. The required diversity of MCSs exploiting such classification models can be achieved by using two techniques, the Bayesian model averaging and the randomised DT ensemble. Both techniques have revealed promising results when applied to real-world problems. In this paper we experimentally compare the classification uncertainty of the Bayesian model averaging with a restarting strategy and the randomised DT ensemble on a synthetic dataset and some domain problems commonly used in the machine learning community. To make the Bayesian DT averaging feasible, we use a Markov Chain Monte Carlo technique. The classification uncertainty is evaluated within an Uncertainty Envelope technique dealing with the class posterior distribution and a given confidence probability. Exploring a full posterior distribution, this technique produces realistic estimates which can be easily interpreted in statistical terms. In our experiments we found out that the Bayesian DTs are superior to the randomised DT ensembles within the Uncertainty Envelope technique.