A Dataset-Level Geometric Framework for Ensemble Classifiers

TL;DR

This paper introduces a geometric framework to analyze ensemble classifiers, providing formulas and theorems that clarify how base classifier performance and diversity influence ensemble effectiveness.

Contribution

It presents a dataset-level geometric approach to formally analyze majority voting schemes, offering new insights into ensemble properties and performance prediction.

Findings

Ensemble performance can be directly calculated using a formula based on performance and dissimilarity.

The number of classifiers affects ensemble schemes differently, with theoretical and empirical validation.

The framework aids in selecting efficient, effective ensembles with fewer classifiers.

Abstract

Ensemble classifiers have been investigated by many in the artificial intelligence and machine learning community. Majority voting and weighted majority voting are two commonly used combination schemes in ensemble learning. However, understanding of them is incomplete at best, with some properties even misunderstood. In this paper, we present a group of properties of these two schemes formally under a dataset-level geometric framework. Two key factors, every component base classifier's performance and dissimilarity between each pair of component classifiers are evaluated by the same metric - the Euclidean distance. Consequently, ensembling becomes a deterministic problem and the performance of an ensemble can be calculated directly by a formula. We prove several theorems of interest and explain their implications for ensembles. In particular, we compare and contrast the effect of the number of component classifiers on these two types of ensemble schemes. Empirical investigation is also conducted to verify the theoretical results when other metrics such as accuracy are used. We believe that the results from this paper are very useful for us to understand the fundamental properties of these two combination schemes and the principles of ensemble classifiers in general. The results are also helpful for us to investigate some issues in ensemble classifiers, such as ensemble performance prediction, selecting a small number of base classifiers to obtain efficient and effective ensembles.