Scalable Ensemble Diversification for OOD Generalization and Detection

TL;DR

This paper introduces a scalable ensemble diversification method that improves out-of-distribution generalization and detection on large-scale datasets like ImageNet by identifying hard samples during training and encouraging model disagreement.

Contribution

The work presents a novel scalable ensemble diversification technique that does not require OOD samples and efficiently encourages disagreement on hard samples, enhancing OOD tasks.

Findings

Significant improvements in OOD generalization on ImageNet.

Enhanced OOD detection accuracy surpassing baseline methods.

Effective disagreement-based uncertainty scoring method.

Abstract

Training a diverse ensemble of models has several practical applications such as providing candidates for model selection with better out-of-distribution (OOD) generalization, and enabling the detection of OOD samples via Bayesian principles. An existing approach to diverse ensemble training encourages the models to disagree on provided OOD samples. However, the approach is computationally expensive and it requires well-separated ID and OOD examples, such that it has only been demonstrated in small-scale settings. $\textbf{Method.}$ This work presents a method for Scalable Ensemble Diversification (SED) applicable to large-scale settings (e.g. ImageNet) that does not require OOD samples. Instead, SED identifies hard training samples on the fly and encourages the ensemble members to disagree on these. To improve scaling, we show how to avoid the expensive computations in existing methods of exhaustive pairwise disagreements across models. $\textbf{Results.}$ We evaluate the benefits of diversification with experiments on ImageNet. First, for OOD generalization, we observe large benefits from the diversification in multiple settings including output-space (classical) ensembles and weight-space ensembles (model soups). Second, for OOD detection, we turn the diversity of ensemble hypotheses into a novel uncertainty score estimator that surpasses a large number of OOD detection baselines. Code is available here: https://github.com/AlexanderRubinstein/diverse-universe-public.