Combining Predictions under Uncertainty: The Case of Random Decision Trees

TL;DR

This paper explores various methods for combining probabilistic predictions from ensembles of decision trees, emphasizing the importance of accounting for uncertainty and evidence accumulation, with experiments showing averaging probabilities is surprisingly effective.

Contribution

The paper introduces and evaluates alternative prediction combination methods inspired by probability, belief functions, and evidence accumulation, demonstrating their effectiveness over simple averaging.

Findings

Averaging probabilities is hard to outperform.

Evidence accumulation improves prediction accuracy in most cases.

Random decision trees provide high diversity for testing combination methods.

Abstract

A common approach to aggregate classification estimates in an ensemble of decision trees is to either use voting or to average the probabilities for each class. The latter takes uncertainty into account, but not the reliability of the uncertainty estimates (so to say, the "uncertainty about the uncertainty"). More generally, much remains unknown about how to best combine probabilistic estimates from multiple sources. In this paper, we investigate a number of alternative prediction methods. Our methods are inspired by the theories of probability, belief functions and reliable classification, as well as a principle that we call evidence accumulation. Our experiments on a variety of data sets are based on random decision trees which guarantees a high diversity in the predictions to be combined. Somewhat unexpectedly, we found that taking the average over the probabilities is actually hard to beat. However, evidence accumulation showed consistently better results on all but very small leafs.