How many classifiers do we need?

TL;DR

This paper analyzes how classifier disagreement and polarization influence ensemble performance, introducing a neural polarization law and providing bounds and asymptotic behaviors to guide ensemble size decisions.

Contribution

It introduces the concept of polarization in classifier ensembles, derives bounds on polarization and disagreement, and links these to ensemble performance and size predictions.

Findings

Polarization is nearly constant across hyperparameters and architectures.

Disagreement correlates linearly with the target under entropy conditions.

Asymptotic analysis predicts ensemble performance from smaller ensembles.

Abstract

As performance gains through scaling data and/or model size experience diminishing returns, it is becoming increasingly popular to turn to ensembling, where the predictions of multiple models are combined to improve accuracy. In this paper, we provide a detailed analysis of how the disagreement and the polarization (a notion we introduce and define in this paper) among classifiers relate to the performance gain achieved by aggregating individual classifiers, for majority vote strategies in classification tasks. We address these questions in the following ways. (1) An upper bound for polarization is derived, and we propose what we call a neural polarization law: most interpolating neural network models are 4/3-polarized. Our empirical results not only support this conjecture but also show that polarization is nearly constant for a dataset, regardless of hyperparameters or architectures of classifiers. (2) The error of the majority vote classifier is considered under restricted entropy conditions, and we present a tight upper bound that indicates that the disagreement is linearly correlated with the target, and that the slope is linear in the polarization. (3) We prove results for the asymptotic behavior of the disagreement in terms of the number of classifiers, which we show can help in predicting the performance for a larger number of classifiers from that of a smaller number. Our theories and claims are supported by empirical results on several image classification tasks with various types of neural networks.