ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization

TL;DR

ZooD introduces a novel method for ranking and ensemble of pre-trained models to enhance out-of-distribution generalization, achieving significant improvements in accuracy and efficiency over existing approaches.

Contribution

The paper proposes ZooD, a new paradigm that ranks and ensembles pre-trained models using feature selection and a variational EM algorithm for OoD tasks.

Findings

Model ranking correlates better with fine-tuning than previous methods.

ZooD achieves up to 9859x faster ranking than brute-force fine-tuning.

Outperforms state-of-the-art methods on diverse OoD tasks, improving accuracy on DomainNet from 46.5% to 50.6%.

Abstract

Recent advances on large-scale pre-training have shown great potentials of leveraging a large set of Pre-Trained Models (PTMs) for improving Out-of-Distribution (OoD) generalization, for which the goal is to perform well on possible unseen domains after fine-tuning on multiple training domains. However, maximally exploiting a zoo of PTMs is challenging since fine-tuning all possible combinations of PTMs is computationally prohibitive while accurate selection of PTMs requires tackling the possible data distribution shift for OoD tasks. In this work, we propose ZooD, a paradigm for PTMs ranking and ensemble with feature selection. Our proposed metric ranks PTMs by quantifying inter-class discriminability and inter-domain stability of the features extracted by the PTMs in a leave-one-domain-out cross-validation manner. The top-K ranked models are then aggregated for the target OoD task. To avoid accumulating noise induced by model ensemble, we propose an efficient variational EM algorithm to select informative features. We evaluate our paradigm on a diverse model zoo consisting of 35 models for various OoD tasks and demonstrate: (i) model ranking is better correlated with fine-tuning ranking than previous methods and up to 9859x faster than brute-force fine-tuning; (ii) OoD generalization after model ensemble with feature selection outperforms the state-of-the-art methods and the accuracy on most challenging task DomainNet is improved from 46.5\% to 50.6\%. Furthermore, we provide the fine-tuning results of 35 PTMs on 7 OoD datasets, hoping to help the research of model zoo and OoD generalization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/zood.