SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models

TL;DR

SSAM introduces a training-free method to merge independently trained multimodal language models by aligning their parameter subspaces, enabling a single model to handle multiple modalities efficiently.

Contribution

The paper presents a novel Singular Subspace Alignment and Merging (SSAM) framework that merges pretrained MLLMs without additional training, addressing modality differences and parameter interference.

Findings

SSAM outperforms prior training-free merging methods.

SSAM surpasses jointly trained multimodal models on four datasets.

The approach is scalable and resource-efficient.

Abstract

Multimodal large language models (MLLMs) achieve strong performance by jointly processing inputs from multiple modalities, such as vision, audio, and language. However, building such models or extending them to new modalities often requires large paired datasets and substantial computational resources. Since many pretrained MLLMs (e.g., vision-language or audio-language) are publicly available, we ask whether we can merge them into a single MLLM that can handle multiple modalities? Merging MLLMs with different input modalities remains challenging, partly because of differences in the learned representations and interference between their parameter spaces. To address these challenges, we propose Singular Subspace Alignment and Merging (SSAM), a training-free model merging framework that unifies independently trained specialist MLLMs into a single model capable of handling any combination of input modalities. SSAM maintains modality-specific parameter updates separately and identifies a shared low-rank subspace for language-related parameter updates, aligns them within this subspace, and merges them to preserve complementary knowledge while minimizing parameter interference. Without using any multimodal training data, SSAM achieves state-of-the-art performance across four datasets, surpassing prior training-free merging methods and even jointly trained multimodal models. These results demonstrate that aligning models in parameter space provides a scalable and resource-efficient alternative to conventional joint multimodal training.