Ensemble Pruning via Margin Maximization

TL;DR

This paper introduces a novel ensemble pruning method that optimizes margin distribution and diversity, resulting in smaller, more efficient models with maintained or improved accuracy, tested on synthetic and real datasets.

Contribution

The paper presents a new ensemble pruning algorithm that enhances diversity and margin maximization, reducing ensemble size while preserving or improving predictive performance.

Findings

Pruned ensembles use fewer classifiers with comparable accuracy.

The method outperforms existing pruning techniques in tests.

Ensembles become more interpretable and computationally efficient.

Abstract

Ensemble models refer to methods that combine a typically large number of classifiers into a compound prediction. The output of an ensemble method is the result of fitting a base-learning algorithm to a given data set, and obtaining diverse answers by reweighting the observations or by resampling them using a given probabilistic selection. A key challenge of using ensembles in large-scale multidimensional data lies in the complexity and the computational burden associated with them. The models created by ensembles are often difficult, if not impossible, to interpret and their implementation requires more computational power than single classifiers. Recent research effort in the field has concentrated in reducing ensemble size, while maintaining their predictive accuracy. We propose a method to prune an ensemble solution by optimizing its margin distribution, while increasing its diversity. The proposed algorithm results in an ensemble that uses only a fraction of the original classifiers, with improved or similar generalization performance. We analyze and test our method on both synthetic and real data sets. The simulations show that the proposed method compares favorably to the original ensemble solutions and to other existing ensemble pruning methodologies.