Regularized Neural Ensemblers

TL;DR

This paper introduces regularized neural networks for ensembling, which adaptively combine diverse models, reducing overfitting and improving generalization across multiple data modalities.

Contribution

It proposes a novel regularization technique for neural ensemblers that encourages diversity and dynamic weighting, outperforming traditional ensemble methods.

Findings

Regularized neural ensemblers improve generalization.

The method reduces overfitting compared to standard ensembling.

Competitive results across vision, NLP, and tabular data.

Abstract

Ensemble methods are known for enhancing the accuracy and robustness of machine learning models by combining multiple base learners. However, standard approaches like greedy or random ensembling often fall short, as they assume a constant weight across samples for the ensemble members. This can limit expressiveness and hinder performance when aggregating the ensemble predictions. In this study, we explore employing regularized neural networks as ensemble methods, emphasizing the significance of dynamic ensembling to leverage diverse model predictions adaptively. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the ensembling model by randomly dropping base model predictions during the training. We demonstrate this approach provides lower bounds for the diversity within the ensemble, reducing overfitting and improving generalization capabilities. Our experiments showcase that the regularized neural ensemblers yield competitive results compared to strong baselines across several modalities such as computer vision, natural language processing, and tabular data.