Differentiable Model Selection for Ensemble Learning

TL;DR

This paper introduces a differentiable model selection framework for ensemble learning that optimizes the selection of models for individual inputs, improving accuracy and robustness across diverse tasks.

Contribution

It presents a novel end-to-end differentiable framework that integrates model selection into ensemble learning, surpassing traditional consensus methods.

Findings

Outperforms conventional ensemble methods in accuracy.

Demonstrates versatility across multiple tasks.

Effective in various learning settings.

Abstract

Model selection is a strategy aimed at creating accurate and robust models. A key challenge in designing these algorithms is identifying the optimal model for classifying any particular input sample. This paper addresses this challenge and proposes a novel framework for differentiable model selection integrating machine learning and combinatorial optimization. The framework is tailored for ensemble learning, a strategy that combines the outputs of individually pre-trained models, and learns to select appropriate ensemble members for a particular input sample by transforming the ensemble learning task into a differentiable selection program trained end-to-end within the ensemble learning model. Tested on various tasks, the proposed framework demonstrates its versatility and effectiveness, outperforming conventional and advanced consensus rules across a variety of settings and learning tasks.