Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging

TL;DR

This paper introduces a training-free, projection-based method for sequentially merging deep models without retraining, reducing memory use and interference, and improving accuracy on CLIP-ViT models.

Contribution

It presents a novel continual merging technique that operates without retraining, using orthogonal projections and adaptive scaling to efficiently combine models sequentially.

Findings

Achieves 5-8% average accuracy improvement on CLIP-ViT models.

Maintains constant memory complexity regardless of the number of models.

Reduces task interference through orthogonal projections.

Abstract

Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their specialized capabilities across different tasks and domains. Current model merging techniques focus on merging all available models simultaneously, with weight interpolation-based methods being the predominant approaches. However, these conventional approaches are not well-suited for scenarios where models become available sequentially, and they often suffer from high memory requirements and potential interference between tasks. In this study, we propose a training-free projection-based continual merging method that processes models sequentially through orthogonal projections of weight matrices and adaptive scaling mechanisms. Our method operates by projecting new parameter updates onto subspaces orthogonal to existing merged parameter updates while using an adaptive scaling mechanism to maintain stable parameter distances, enabling efficient sequential integration of task-specific knowledge. Our approach maintains constant memory complexity to the number of models, minimizes interference between tasks through orthogonal projections, and retains the performance of previously merged models through adaptive task vector scaling. Extensive experiments on CLIP-ViT models demonstrate that our method achieves a 5-8% average accuracy improvement while maintaining robust performance in different task orderings.