Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection

TL;DR

This paper introduces Joint Parameter Selection (JPS), a method for fine-tuning pre-trained models on limited source domains that preserves their ability to generalize to unseen domains by selectively updating a sparse set of parameters.

Contribution

The paper proposes JPS, a novel parameter-efficient fine-tuning method with a theoretical generalization bound and a practical selection mechanism for improved domain generalization.

Findings

JPS outperforms state-of-the-art domain generalization methods.

Theoretical analysis supports the effectiveness of sparse parameter updates.

JPS maintains model generalization while achieving high task performance.

Abstract

Domain generalization seeks to develop models trained on a limited set of source domains that are capable of generalizing effectively to unseen target domains. While the predominant approach leverages large-scale pre-trained vision models as initialization, recent studies have highlighted that full fine-tuning can compromise the intrinsic generalization capabilities of these models. To address this limitation, parameter-efficient adaptation strategies have emerged, wherein only a subset of model parameters is selectively fine-tuned, thereby balancing task adaptation with the preservation of generalization. Motivated by this paradigm, we introduce Joint Parameter Selection (JPS), a novel method that restricts updates to a small, sparse subset of parameters, thereby retaining and harnessing the generalization strength of pre-trained models. Theoretically, we establish a generalization error bound that explicitly accounts for the sparsity of parameter updates, thereby providing a principled justification for selective fine-tuning. Practically, we design a selection mechanism employing dual operators to identify and update parameters exhibiting consistent and significant gradients across all source domains. Extensive benchmark experiments demonstrate that JPS achieves superior performance compared to state-of-the-art domain generalization methods, substantiating both the efficiency and efficacy of the proposed approach.