Boosting Revisited: Benchmarking and Advancing LP-Based Ensemble Methods

TL;DR

This paper provides a comprehensive empirical evaluation of LP-based boosting methods, introducing two new algorithms, and compares their performance to existing state-of-the-art methods across various metrics and datasets.

Contribution

It presents the first large-scale experimental study of LP-based boosting, introduces two novel methods, and analyzes their performance and properties in detail.

Findings

LP-based boosting can outperform or match XGBoost and LightGBM with shallow trees.

Totally corrective methods produce sparser ensembles than heuristics.

Pre-trained ensembles can be thinned without performance loss.

Abstract

Despite their theoretical appeal, totally corrective boosting methods based on linear programming have received limited empirical attention. In this paper, we conduct the first large-scale experimental study of six LP-based boosting formulations, including two novel methods, NM-Boost and QRLP-Boost, across 20 diverse datasets. We evaluate the use of both heuristic and optimal base learners within these formulations, and analyze not only accuracy, but also ensemble sparsity, margin distribution, anytime performance, and hyperparameter sensitivity. We show that totally corrective methods can outperform or match state-of-the-art heuristics like XGBoost and LightGBM when using shallow trees, while producing significantly sparser ensembles. We further show that these methods can thin pre-trained ensembles without sacrificing performance, and we highlight both the strengths and limitations of using optimal decision trees in this context.