One Size Does Not Fit All: A Distribution-Aware Sparsification for More Precise Model Merging

TL;DR

This paper introduces TADrop, an adaptive sparsification method that assigns different pruning levels to model parameters based on their distribution, significantly improving multi-task model merging performance.

Contribution

TADrop is a novel, distribution-aware sparsification strategy that dynamically adjusts pruning ratios per tensor, outperforming uniform sparsification in model merging tasks.

Findings

TADrop improves merging performance by an average of 2.0% across 8 ViT tasks.

It effectively preserves critical parameters while aggressively pruning redundant ones.

TADrop is compatible with various merging methods and models.

Abstract

Model merging has emerged as a compelling data-free paradigm for multi-task learning, enabling the fusion of multiple fine-tuned models into a single, powerful entity. A key technique in merging methods is sparsification, which prunes redundant parameters from task vectors to mitigate interference. However, prevailing approaches employ a ``one-size-fits-all'' strategy, applying a uniform sparsity ratio that overlooks the inherent structural and statistical heterogeneity of model parameters. This often leads to a suboptimal trade-off, where critical parameters are inadvertently pruned while less useful ones are retained. To address this limitation, we introduce \textbf{TADrop} (\textbf{T}ensor-wise \textbf{A}daptive \textbf{Drop}), an adaptive sparsification strategy that respects this heterogeneity. Instead of a global ratio, TADrop assigns a tailored sparsity level to each parameter tensor based on its distributional properties. The core intuition is that tensors with denser, more redundant distributions can be pruned aggressively, while sparser, more critical ones are preserved. As a simple and plug-and-play module, we validate TADrop by integrating it with foundational, classic, and SOTA merging methods. Extensive experiments across diverse tasks (vision, language, and multimodal) and models (ViT, BEiT) demonstrate that TADrop consistently and significantly boosts their performance. For instance, when enhancing a leading merging method, it achieves an average performance gain of 2.0\% across 8 ViT-B/32 tasks. TADrop provides a more effective way to mitigate parameter interference by tailoring sparsification to the model's structure, offering a new baseline for high-performance model merging.