Domain-Adaptive Model Merging Across Disconnected Modes

TL;DR

This paper introduces DMM, a novel data-free model merging framework that effectively combines highly divergent models across different domains while preserving critical knowledge.

Contribution

DMM is the first framework to handle highly divergent models through a three-step process involving model similarity merging and pseudo-data distillation.

Findings

DMM achieves state-of-the-art performance on various benchmarks.

It effectively merges models with high divergence without data sharing.

DMM preserves rare and critical knowledge during merging.

Abstract

Learning across domains is challenging when data cannot be centralized due to privacy or heterogeneity, which limits the ability to train a single comprehensive model. Model merging provides an appealing alternative by consolidating knowledge from multiple specialized models into one, avoiding data sharing and reducing retraining cost. In this work, we present DMM, a data-free model merging framework designed to handle highly divergent models. DMM proceeds in three steps. First, domain-specific models are trained independently. Second, models with high similarity are merged using standard techniques to ensure stability. Third, we synthesize pseudo-data from normalization statistics and distill knowledge from divergent models into the merged model through a lightweight refinement guided by these samples. This approach preserves rare but critical knowledge while maintaining stability. Extensive experiments on unimodal and multimodal benchmarks show that DMM achieves state-of-the-art performance over existing merging methods.