SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning

TL;DR

This paper introduces SoMA, a novel method that selectively fine-tunes minor singular value components of pre-trained models to enhance domain generalization without extra inference costs.

Contribution

The paper proposes Singular Value Decomposed Minor Components Adaptation (SoMA), a new approach that improves domain generalization by tuning minor singular components while preserving pre-trained weights.

Findings

Achieves state-of-the-art results on domain generalization benchmarks.

Maintains compatibility with any backbone or head without additional inference overhead.

Effectively balances generalization and discriminability through selective tuning and annealing strategies.

Abstract

Domain generalization (DG) aims to adapt a model using one or multiple source domains to ensure robust performance in unseen target domains. Recently, Parameter-Efficient Fine-Tuning (PEFT) of foundation models has shown promising results in the context of DG problem. Nevertheless, existing PEFT methods still struggle to strike a balance between preserving generalizable components of the pre-trained model and learning task-specific features. To gain insights into the distribution of generalizable components, we begin by analyzing the pre-trained weights through the lens of singular value decomposition. Building on these insights, we introduce Singular Value Decomposed Minor Components Adaptation (SoMA), an approach that selectively tunes minor singular components while keeping the residual parts frozen. SoMA effectively retains the generalization ability of the pre-trained model while efficiently acquiring task-specific skills. Moreover, we freeze domain-generalizable blocks and employ an annealing weight decay strategy, thereby achieving an optimal balance in the delicate trade-off between generalizability and discriminability. SoMA attains state-of-the-art results on multiple benchmarks that span both domain generalized semantic segmentation to domain generalized object detection. In addition, our methods introduce no additional inference overhead or regularization loss, maintain compatibility with any backbone or head, and are designed to be versatile, allowing easy integration into a wide range of tasks.