Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

TL;DR

This paper explores advanced strategies for creating diverse and effective ensembles of deep neural networks, introducing novel methods like TreeNets and ensemble-aware training to improve performance beyond traditional approaches.

Contribution

It systematically compares ensembling strategies and proposes new methods such as TreeNets and diversity-encouraging losses for end-to-end training of ensembles.

Findings

TreeNets improve ensemble performance

Diverse ensembles trained end-to-end outperform classical ensembles

Ensemble-aware losses increase oracle accuracy

Abstract

Convolutional Neural Networks have achieved state-of-the-art performance on a wide range of tasks. Most benchmarks are led by ensembles of these powerful learners, but ensembling is typically treated as a post-hoc procedure implemented by averaging independently trained models with model variation induced by bagging or random initialization. In this paper, we rigorously treat ensembling as a first-class problem to explicitly address the question: what are the best strategies to create an ensemble? We first compare a large number of ensembling strategies, and then propose and evaluate novel strategies, such as parameter sharing (through a new family of models we call TreeNets) as well as training under ensemble-aware and diversity-encouraging losses. We demonstrate that TreeNets can improve ensemble performance and that diverse ensembles can be trained end-to-end under a unified loss, achieving significantly higher "oracle" accuracies than classical ensembles.