Model Merging in the Essential Subspace

TL;DR

This paper introduces ESM, a novel model merging framework that uses PCA to identify essential subspaces, reducing task interference and effectively combining multiple fine-tuned models into a single multi-task model.

Contribution

The paper proposes a new model merging method leveraging PCA-based essential subspaces and polarized scaling to improve multi-task model integration without additional training.

Findings

Achieves state-of-the-art performance in multi-task model merging.

Effectively reduces task interference during model merging.

Preserves core task-specific knowledge in merged models.

Abstract

Model merging aims to integrate multiple task-specific fine-tuned models derived from a shared pre-trained checkpoint into a single multi-task model without additional training. Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models. In this paper, we propose ESM (Essential Subspace Merging) , a robust framework for effective model merging. We begin by performing Principal Component Analysis (PCA) on feature shifts induced by parameter updates. The resulting principal directions span an essential subspace that dominantly influences feature representations. Each task's parameter update matrix is projected onto its respective essential subspace for low-rank decomposition before merging. This methodology mitigates inter-task interference while preserving core task-specific functionality. Furthermore, we introduce a multi-level polarized scaling strategy that amplifies parameters containing critical knowledge and suppresses redundant ones, preventing essential knowledge from being overwhelmed during fusion. Extensive experiments across multiple task sets and model scales demonstrate that our method achieves state-of-the-art performance in multi-task model merging.