Training-Free Model Merging for Multi-target Domain Adaptation

TL;DR

This paper introduces a training-free approach to merge models adapted to different target domains for scene understanding, addressing data privacy and bandwidth constraints, and achieving performance comparable to data-dependent methods.

Contribution

It proposes a novel method for merging models and their normalization statistics without access to training data, using empirical mode connectivity analysis and Gaussian modeling.

Findings

Linear model merging suffices with shared backbone weights.

The method achieves comparable performance to data-dependent baselines.

It effectively bypasses data privacy and bandwidth issues.

Abstract

In this paper, we study multi-target domain adaptation of scene understanding models. While previous methods achieved commendable results through inter-domain consistency losses, they often assumed unrealistic simultaneous access to images from all target domains, overlooking constraints such as data transfer bandwidth limitations and data privacy concerns. Given these challenges, we pose the question: How to merge models adapted independently on distinct domains while bypassing the need for direct access to training data? Our solution to this problem involves two components, merging model parameters and merging model buffers (i.e., normalization layer statistics). For merging model parameters, empirical analyses of mode connectivity surprisingly reveal that linear merging suffices when employing the same pretrained backbone weights for adapting separate models. For merging model buffers, we model the real-world distribution with a Gaussian prior and estimate new statistics from the buffers of separately trained models. Our method is simple yet effective, achieving comparable performance with data combination training baselines, while eliminating the need for accessing training data. Project page: https://air-discover.github.io/ModelMerging