Ensemble Validation: Selectivity has a Price, but Variety is Free

TL;DR

This paper analyzes the trade-offs in ensemble classifier selection, showing that increasing variety in hypotheses does not incur additional error penalties if the same fraction is used, balancing selectivity and diversity.

Contribution

It introduces an error bound framework for ensembles that balances selectivity and hypothesis set richness without penalty.

Findings

Ensemble error bounds depend on selectivity and hypothesis set size.

Using more classifiers in the ensemble does not increase error if the same fraction is selected.

Richer hypothesis sets can be used freely without penalty.

Abstract

Suppose some classifiers are selected from a set of hypothesis classifiers to form an equally-weighted ensemble that selects a member classifier at random for each input example. Then the ensemble has an error bound consisting of the average error bound for the member classifiers, a term for selectivity that varies from zero (if all hypothesis classifiers are selected) to a standard uniform error bound (if only a single classifier is selected), and small constants. There is no penalty for using a richer hypothesis set if the same fraction of the hypothesis classifiers are selected for the ensemble.