DC-Merge: Improving Model Merging with Directional Consistency

TL;DR

DC-Merge introduces a novel approach to model merging that maintains directional consistency by balancing energy distribution and aligning task vectors, leading to improved multi-task model integration.

Contribution

It proposes a new method, DC-Merge, that enhances model merging by addressing energy imbalance and geometric inconsistency in task vectors.

Findings

Achieves state-of-the-art results on vision benchmarks.

Effective in both full fine-tuning and LoRA settings.

Improves knowledge retention in multi-task models.

Abstract

Model merging aims to integrate multiple task-adapted models into a unified model that preserves the knowledge of each task. In this paper, we identify that the key to this knowledge retention lies in maintaining the directional consistency of singular spaces between merged multi-task vector and individual task vectors. However, this consistency is frequently compromised by two issues: i) an imbalanced energy distribution within task vectors, where a small fraction of singular values dominate the total energy, leading to the neglect of semantically important but weaker components upon merging, and ii) the geometric inconsistency of task vectors in parameter space, which causes direct merging to distort their underlying directional geometry. To address these challenges, we propose DC-Merge, a method for directional-consistent model merging. It first balances the energy distribution of each task vector by smoothing its singular values, ensuring all knowledge components are adequately represented. These energy-balanced vectors are then projected onto a shared orthogonal subspace to align their directional geometries with minimal reconstruction error. Finally, the aligned vectors are aggregated in the shared orthogonal subspace and projected back to the original parameter space. Extensive experiments on vision and vision-language benchmarks show that DC-Merge consistently achieves state-of-the-art performance in both full fine-tuning and LoRA settings. The implementation code is available at https://github.com/Tobeginwith/DC-Merge.