PSEO: Optimizing Post-hoc Stacking Ensemble Through Hyperparameter Tuning

TL;DR

PSEO is a novel framework that optimizes post-hoc stacking ensembles in AutoML by hyperparameter tuning, leading to improved ensemble performance across diverse datasets.

Contribution

It introduces a hyperparameter search space for post-hoc stacking and employs binary quadratic programming for base model selection, enhancing ensemble adaptability.

Findings

Achieves the best average test rank among 16 methods.

Outperforms recent AutoML post-hoc ensemble strategies.

Demonstrates effectiveness on 80 public datasets.

Abstract

The Combined Algorithm Selection and Hyperparameter Optimization (CASH) problem is fundamental in Automated Machine Learning (AutoML). Inspired by the success of ensemble learning, recent AutoML systems construct post-hoc ensembles for final predictions rather than relying on the best single model. However, while most CASH methods conduct extensive searches for the optimal single model, they typically employ fixed strategies during the ensemble phase that fail to adapt to specific task characteristics. To tackle this issue, we propose PSEO, a framework for post-hoc stacking ensemble optimization. First, we conduct base model selection through binary quadratic programming, with a trade-off between diversity and performance. Furthermore, we introduce two mechanisms to fully realize the potential of multi-layer stacking. Finally, PSEO builds a hyperparameter space and searches for the optimal post-hoc ensemble strategy within it. Empirical results on 80 public datasets show that \sys achieves the best average test rank (2.96) among 16 methods, including post-hoc designs in recent AutoML systems and state-of-the-art ensemble learning methods.