When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging

TL;DR

This paper identifies a spectral over-accumulation problem in model merging when shared knowledge is over-counted, and introduces Singular Value Calibration (SVC), a simple post-processing method that improves merging performance across vision and language tasks.

Contribution

The paper reveals spectral over-accumulation in model merging and proposes SVC, a data-free method to correct singular value inflation, enhancing merging accuracy.

Findings

SVC improves merging performance across vision and language benchmarks.

SVC achieves state-of-the-art results in model merging tasks.

Modifying singular values alone enhances Task Arithmetic by 13.0%.

Abstract

Model merging combines multiple fine-tuned models into a single model by adding their weight updates, providing a lightweight alternative to retraining. Existing methods primarily target resolving conflicts between task updates, leaving the failure mode of over-counting shared knowledge unaddressed. We show that when tasks share aligned spectral directions (i.e., overlapping singular vectors), a simple linear combination repeatedly accumulates these directions, inflating the singular values and biasing the merged model toward shared subspaces. To mitigate this issue, we propose Singular Value Calibration (SVC), a training-free and data-free post-processing method that quantifies subspace overlap and rescales inflated singular values to restore a balanced spectrum. Across vision and language benchmarks, SVC consistently improves strong merging baselines and achieves state-of-the-art performance. Furthermore, by modifying only the singular values, SVC improves the performance of Task Arithmetic by 13.0%. Code is available at https://github.com/lyymuwu/SVC.