Bridging Domains through Subspace-Aware Model Merging

TL;DR

This paper introduces SCORE, a novel model merging method that reduces subspace conflicts to improve domain generalization, outperforming existing approaches across various architectures and scales.

Contribution

The paper proposes SCORE, a subspace conflict-resolving merging technique that enhances domain generalization by aligning task-specific models into a shared orthogonal basis.

Findings

SCORE outperforms existing merging methods in domain generalization tasks.

Merging models trained on different domains induces stronger subspace conflicts.

SCORE effectively mitigates singular subspace conflicts, improving generalization.

Abstract

Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or multi-task scenarios, but domain generalization in model merging remains underexplored. We investigate how merging models fine-tuned on distinct domains affects generalization to unseen domains. Through an analysis of parameter competition in the task matrix using singular value decomposition, we show that merging models trained under different distribution shifts induces stronger conflicts between their subspaces compared to traditional multi-task settings. To mitigate this issue, we propose SCORE (Subspace COnflict-Resolving mErging), a method designed to alleviate such singular subspace conflicts. SCORE finds a shared orthogonal basis by computing the principal components of the concatenated leading singular vectors of all models. It then projects each task matrix into the shared basis, pruning off-diagonal components to remove conflicting singular directions. SCORE consistently outperforms, on average, existing model merging approaches in domain generalization settings across a variety of architectures and model scales, demonstrating its effectiveness and scalability.