Domain-Aware Fine-Tuning of Foundation Models

TL;DR

This paper introduces Domino, a domain-aware normalization method for foundation models that improves their robustness and adaptability to unseen domains in computer vision tasks.

Contribution

It proposes a novel domain adaptive normalization technique called Domino that explicitly incorporates domain embeddings during fine-tuning.

Findings

Domino enhances model robustness across unseen domains.

Domain-aware components improve zero-shot domain adaptation.

Comparison shows benefits over traditional fine-tuning methods.

Abstract

Foundation models (FMs) have revolutionized computer vision, enabling effective learning across different domains. However, their performance under domain shift is yet underexplored. This paper investigates the zero-shot domain adaptation potential of FMs by comparing different backbone architectures and introducing novel domain-aware components that leverage domain related textual embeddings. We propose domain adaptive normalization, termed as Domino, which explicitly leverages domain embeddings during fine-tuning, thus making the model domain aware. Ultimately, Domino enables more robust computer vision models that can adapt effectively to various unseen domains.