Performance-Agnostic Fusion of Probabilistic Classifier Outputs

TL;DR

This paper introduces a novel method for combining probabilistic classifier outputs that accounts for diversity among classifiers, iteratively reaching a consensus without relying on prior information or calibration.

Contribution

The proposed approach uniquely considers classifier diversity and iteratively updates predictions to achieve consensus, unlike traditional equal-weight methods.

Findings

Effective on benchmark datasets for accuracy-focused tasks

Outperforms traditional methods in diverse classifier scenarios

Does not produce calibrated probabilities for further probabilistic use

Abstract

We propose a method for combining probabilistic outputs of classifiers to make a single consensus class prediction when no further information about the individual classifiers is available, beyond that they have been trained for the same task. The lack of relevant prior information rules out typical applications of Bayesian or Dempster-Shafer methods, and the default approach here would be methods based on the principle of indifference, such as the sum or product rule, which essentially weight all classifiers equally. In contrast, our approach considers the diversity between the outputs of the various classifiers, iteratively updating predictions based on their correspondence with other predictions until the predictions converge to a consensus decision. The intuition behind this approach is that classifiers trained for the same task should typically exhibit regularities in their outputs on a new task; the predictions of classifiers which differ significantly from those of others are thus given less credence using our approach. The approach implicitly assumes a symmetric loss function, in that the relative cost of various prediction errors are not taken into account. Performance of the model is demonstrated on different benchmark datasets. Our proposed method works well in situations where accuracy is the performance metric; however, it does not output calibrated probabilities, so it is not suitable in situations where such probabilities are required for further processing.