A Boosting Approach to Constructing an Ensemble Stack

TL;DR

This paper introduces a boosting-based method for constructing an ensemble stack that improves interpretability and training efficiency, demonstrating competitive accuracy and simplicity compared to existing algorithms and XGBoost.

Contribution

It presents a novel evolutionary ensemble learning approach using boosting to build a stack of programs, enhancing interpretability and efficiency over prior methods.

Findings

Achieves comparable prediction accuracy to state-of-the-art ensemble algorithms.

Produces simpler models that are easier to interpret.

Outperforms XGBoost in accuracy and efficiency on high-cardinality datasets.

Abstract

An approach to evolutionary ensemble learning for classification is proposed in which boosting is used to construct a stack of programs. Each application of boosting identifies a single champion and a residual dataset, i.e. the training records that thus far were not correctly classified. The next program is only trained against the residual, with the process iterating until some maximum ensemble size or no further residual remains. Training against a residual dataset actively reduces the cost of training. Deploying the ensemble as a stack also means that only one classifier might be necessary to make a prediction, so improving interpretability. Benchmarking studies are conducted to illustrate competitiveness with the prediction accuracy of current state-of-the-art evolutionary ensemble learning algorithms, while providing solutions that are orders of magnitude simpler. Further benchmarking with a high cardinality dataset indicates that the proposed method is also more accurate and efficient than XGBoost.