Model Agnostic Combination for Ensemble Learning

TL;DR

The paper introduces MAC, a model-agnostic ensemble method that optimally combines models, allowing dynamic addition or replacement of sub-models post-deployment without retraining, outperforming traditional ensemble techniques.

Contribution

MAC is a novel ensemble technique that remains invariant to the number of sub-models, enabling flexible model management after deployment.

Findings

MAC outperforms classical averaging methods.

MAC shows competitive results to XGBoost with a fixed number of models.

MAC outperforms XGBoost when adding models without retraining.

Abstract

Ensemble of models is well known to improve single model performance. We present a novel ensembling technique coined MAC that is designed to find the optimal function for combining models while remaining invariant to the number of sub-models involved in the combination. Being agnostic to the number of sub-models enables addition and replacement of sub-models to the combination even after deployment, unlike many of the current methods for ensembling such as stacking, boosting, mixture of experts and super learners that lock the models used for combination during training and therefore need retraining whenever a new model is introduced into the ensemble. We show that on the Kaggle RSNA Intracranial Hemorrhage Detection challenge, MAC outperforms classical average methods, demonstrates competitive results to boosting via XGBoost for a fixed number of sub-models, and outperforms it when adding sub-models to the combination without retraining.