Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters

TL;DR

This paper introduces Soup-Adapters, an ensemble approach that enhances the robustness and performance of foundation models in few-shot domain adaptation by averaging multiple adapters trained with diverse hyperparameters.

Contribution

The paper proposes Soup-Adapters, a novel ensemble method that improves robustness and reduces hyperparameter sensitivity in domain adaptation of foundation models.

Findings

Ensemble of adapters outperforms individual adapters in accuracy and robustness.

Soup-Adapters are less sensitive to residual ratio hyperparameter.

Method is applicable to CLIP and DINOv2 models.

Abstract

In this paper, we tackle two fundamental problems in few-shot domain adaptation of foundation models. First, hyperparameter tuning is often impractical due to the lack of large validation datasets. Second, model robustness under distribution shifts where test time data deviates slightly from training distributions, remains a concern. We show that by training multiple independent adapters and averaging their outputs, the new model has a higher performance and is more robust to distribution shifts compared to any individual adapter. This improvement holds even when the adapters are trained with diverse hyperparameters sampled from a wide range, resulting in varied individual performance. Consequently, our method addresses both of the problems described above. The ensemble is also significantly less sensitive to the residual ratio, a critical hyperparameter of CLIP-Adapter. Since the ensemble can be reparameterized to a single adapter again using a principled concatenation of the parameters, we refer to our method as Soup-Adapter. This is also the first study to explore CLIP adapter-style techniques for DINOv2 and to directly compare them with CLIP in this setting.