Adaptive Forests For Classification

TL;DR

This paper introduces Adaptive Forests, a new classification method that adaptively assigns weights to trees, outperforming traditional ensemble models like RF and XGBoost across multiple datasets.

Contribution

It presents a novel adaptive weighting approach combining OP2T and MIO, improving classification accuracy over existing ensemble methods.

Findings

AF outperforms RF and XGBoost on diverse datasets.

Adaptive weighting improves classification performance.

The method is effective for both binary and multi-class problems.

Abstract

Random Forests (RF) and Extreme Gradient Boosting (XGBoost) are two of the most widely used and highly performing classification and regression models. They aggregate equally weighted CART trees, generated randomly in RF or sequentially in XGBoost. In this paper, we propose Adaptive Forests (AF), a novel approach that adaptively selects the weights of the underlying CART models. AF combines (a) the Optimal Predictive-Policy Trees (OP2T) framework to prescribe tailored, input-dependent unequal weights to trees and (b) Mixed Integer Optimization (MIO) to refine weight candidates dynamically, enhancing overall performance. We demonstrate that AF consistently outperforms RF, XGBoost, and other weighted RF in binary and multi-class classification problems over 20+ real-world datasets.