Domain Generalization Guided by Large-Scale Pre-Trained Priors

TL;DR

This paper proposes FT-LP, a novel fine-tuning method that leverages large-scale pre-trained models as priors throughout the domain generalization process, significantly improving generalization to unseen domains.

Contribution

The paper introduces FT-LP, a new approach that incorporates pre-trained models as priors during fine-tuning for domain generalization, supported by a theoretical error bound and practical implementation strategies.

Findings

FT-LP improves domain generalization performance across multiple datasets.

Theoretical analysis confirms the validity of using pre-trained priors in DG.

Experimental results show significant gains over baseline methods.

Abstract

Domain generalization (DG) aims to train a model from limited source domains, allowing it to generalize to unknown target domains. Typically, DG models only employ large-scale pre-trained models during the initialization of fine-tuning. However, large-scale pre-trained models already possess the ability to resist domain shift. If we reference pre-trained models continuously during fine-tuning to maintain this ability, it could further enhance the generalization ability of the DG model. For this purpose, we introduce a new method called Fine-Tune with Large-scale pre-trained Priors (FT-LP), which incorporates the pre-trained model as a prior into the DG fine-tuning process, ensuring that the model refers to its pre-trained model at each optimization step. FT-LP comprises a theoretical framework and a simple implementation strategy. In theory, we verify the rationality of FT-LP by introducing a generalization error bound with the pre-trained priors for DG. In implementation, we utilize an encoder to simulate the model distribution, enabling the use of FT-LP when only pre-trained weights are available. In summary, we offer a new fine-tuning method for DG algorithms to utilize pre-trained models throughout the fine-tuning process. Through experiments on various datasets and DG models, our proposed method exhibits significant improvements, indicating its effectiveness.