SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation

TL;DR

SyMerge introduces a lightweight, self-labeling framework that optimizes a single task-specific layer to enhance synergy in model merging, achieving state-of-the-art results across multiple domains.

Contribution

It proposes a novel single-layer adaptation method with self-labeling for synergistic model merging, surpassing interference-focused approaches.

Findings

Achieves state-of-the-art performance on vision, dense prediction, and NLP benchmarks.

Adapted layers transfer effectively to other merging methods.

Simple single-layer optimization significantly improves merge quality.

Abstract

Model merging offers an efficient alternative to multi-task learning by combining independently fine-tuned models, but most prior approaches focus mainly on avoiding task interference. We argue instead that the real potential of merging lies in achieving synergy, where tasks enhance one another. Our intuition comes from a pilot study showing that when a classifier trained on one task is paired with the encoder of another, the resulting cross-task performance strongly predicts merge quality. Moreover, adapting even a single task-specific layer can substantially improve this compatibility, suggesting a simple yet powerful lever for synergy. Building on this insight, we introduce SyMerge, a lightweight framework that jointly optimizes one task-specific layer and merging coefficients. To ensure stability without labels, SyMerge employs a robust self-labeling strategy guided by expert model predictions, avoiding the pitfalls of entropy-based adaptation. This minimalist yet principled design achieves state-of-the-art results across vision, dense prediction, and NLP benchmarks, while also producing adapted layers that transfer effectively to other merging methods. Our code is available at https://aim-skku.github.io/SyMerge/