SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging

TL;DR

SE-Merging is a novel framework that enhances dynamic model merging by leveraging task differentiation and adaptive rescaling, achieving improved multi-task performance without extra training.

Contribution

It introduces a self-enhanced merging method that dynamically identifies tasks and adjusts merging coefficients, advancing understanding and capabilities of model merging.

Findings

Significant performance improvements over existing merging methods

Effective task differentiation and adaptive rescaling in merging process

Compatibility with various existing model merging techniques

Abstract

Model merging has gained increasing attention due to its intriguing property: interpolating the parameters of different task-specific fine-tuned models leads to multi-task abilities. However, despite its empirical success, the underlying mechanisms of model merging remain poorly understood. In this work, we delve into the mechanism behind model merging from a representation perspective. Our analysis reveals that model merging achieves multi-task abilities through two key capabilities: i) distinguishing samples from different tasks, and ii) adapting to the corresponding expert model for each sample. These two capabilities allow the merged model to retain task-specific expertise, enabling efficient multi-task adaptation. Building on these insights, we propose \texttt{SE-Merging}, a self-enhanced model merging framework that leverages these two characteristics to dynamically identify the corresponding task for each sample and then adaptively rescales the merging coefficients to further enhance task-specific expertise in the merged model. Notably, \texttt{SE-Merging} achieves dynamic model merging without additional training. Extensive experiments demonstrate that \texttt{SE-Merging} achieves significant performance improvements while remaining compatible with existing model merging techniques.