Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic

TL;DR

Localize-and-Stitch is a novel model merging technique that identifies and reintegrates only essential localized regions from finetuned models, improving performance and efficiency in combining multiple models for vision and language tasks.

Contribution

The paper introduces a localized merging approach that selectively combines critical regions of finetuned models, reducing interference and enhancing model composition efficiency.

Findings

Outperforms existing model merging methods on vision and language benchmarks.

Effectively locates sparse, task-specific regions responsible for finetuned performance.

Enables model compression and continual skill composition with minimal overhead.

Abstract

Model merging offers an effective strategy to combine the strengths of multiple finetuned models into a unified model that preserves the specialized capabilities of each. Existing methods merge models in a global manner, performing arithmetic operations across all model parameters. However, such global merging often leads to task interference, degrading the performance of the merged model. In this work, we introduce Localize-and-Stitch, a novel approach that merges models in a localized way. Our algorithm works in two steps: i) Localization: identify tiny ($1\%$ of the total parameters) localized regions in the finetuned models containing essential skills for the downstream tasks, and ii) Stitching: reintegrate only these essential regions back into the pretrained model for task synergy. We demonstrate that our approach effectively locates sparse regions responsible for finetuned performance, and the localized regions could be treated as compact and interpretable representations of the finetuned models (tasks). Empirically, we evaluate our method on various vision and language benchmarks, showing that it outperforms existing model merging methods under different data availability scenarios. Beyond strong empirical performance, our algorithm also facilitates model compression and preserves pretrained knowledge, enabling flexible and continual skill composition from multiple finetuned models with minimal storage and computational overhead. Our code is available at https://github.com/uiuctml/Localize-and-Stitch.