From Coefficients to Directions: Rethinking Model Merging with Directional Alignment

TL;DR

This paper introduces a novel model merging method that emphasizes aligning directional structures in parameter and feature spaces, improving coherence and performance over traditional coefficient-based approaches.

Contribution

It proposes a unified geometric framework called Merging with Directional Alignment that enhances model merging by focusing on directional consistency.

Findings

Directional alignment improves structural coherence.

The method outperforms traditional merging strategies.

Extensive experiments validate effectiveness across tasks.

Abstract

Model merging has emerged as a practical paradigm for integrating multiple independently trained models into a single model without joint retraining. Previous studies have demonstrated the effectiveness of combining parameters through strategies such as parameter decomposition, coefficient optimization, and subspace learning, significantly reducing the need for expensive joint training and achieving strong empirical performance across diverse tasks. However, these approaches predominantly treat merging as a problem of parameter space decomposition or fusion coefficient optimization, while overlooking the critical role of directional information in both parameter and feature spaces. In practice, na\"ive merging introduces inconsistencies in dominant parameter directions and disrupts structural coherence across models, which can degrade performance. Moreover, coefficient-based optimization methods implicitly assume compatible feature-space directions across models. However, Neural Collapse indicates that class features follow structured directional patterns, which may differ across independently trained models, making coefficient optimization alone insufficient. In this work, we emphasize the importance of \emph{directional alignment} and introduce a unified geometric framework, \emph{Merging with Directional Alignment} (\method{}), which aligns directional structures consistently in both the parameter and feature spaces. Our analysis shows that directional alignment improves structural coherence, and extensive experiments across benchmarks, model scales, and task configurations further validate the effectiveness of our approach.