Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging

TL;DR

This paper introduces Sparse Complementary Fusion with reverse KL (SCF-RKL), a novel model merging method that improves stability and generalization by controlling functional interference through sparse, distribution-aware updates, outperforming existing methods across diverse benchmarks.

Contribution

The paper presents SCF-RKL, a new model merging framework that explicitly measures and minimizes functional divergence using reverse KL divergence, enabling more stable and effective model integration.

Findings

SCF-RKL outperforms existing merging methods on 24 benchmarks.

It maintains stable representations while integrating new capabilities.

The approach improves generalization and generation stability.

Abstract

Model merging has emerged as a promising paradigm for composing the capabilities of large language models by directly operating in weight space, enabling the integration of specialized models without costly retraining. However, existing merging methods largely rely on parameter-space heuristics, which often introduce severe interference, leading to degraded generalization and unstable generation behaviors such as repetition and incoherent outputs. In this work, we propose Sparse Complementary Fusion with reverse KL (SCF-RKL), a novel model merging framework that explicitly controls functional interference through sparse, distribution-aware updates. Instead of assuming linear additivity in parameter space, SCF-RKL measures the functional divergence between models using reverse Kullback-Leibler divergence and selectively incorporates complementary parameters. This mode-seeking, sparsity-inducing design effectively preserves stable representations while integrating new capabilities. We evaluate SCF-RKL across a wide range of model scales and architectures, covering both reasoning-focused and instruction-tuned models. Extensive experiments on 24 benchmarks spanning advanced reasoning, general reasoning and knowledge, instruction following, and safety demonstrate, vision classification that SCF-RKL consistently outperforms existing model merging methods while maintaining strong generalization and generation stability.