Less is More: Efficient Model Merging with Binary Task Switch

TL;DR

This paper introduces T-Switch, a method for efficient multi-task model merging that uses binarized task vectors to reduce storage and conflict, achieving comparable or better performance with minimal storage overhead.

Contribution

The paper proposes a novel binarization approach for task vectors and introduces T-Switch and Auto-Switch for efficient, storage-friendly multi-task model merging.

Findings

Binarized task vectors retain performance with minimal loss.

T-Switch reduces storage to 1-3% of full-precision parameters.

Auto-Switch enables training-free switch combination.

Abstract

As an effective approach to equip models with multi-task capabilities without additional training, model merging has garnered significant attention. However, existing methods face challenges of redundant parameter conflicts and the excessive storage burden of parameters. In this work, through controlled experiments, we reveal that for task vectors, only those parameters with magnitudes above a certain threshold contribute positively to the task, exhibiting a pulse-like characteristic. We then attempt leveraging this characteristic to binarize the task vectors and reduce storage overhead. Further controlled experiments show that the binarized task vectors incur almost no decrease in fine-tuning and merging performance, and even exhibit stronger performance improvements as the proportion of redundant parameters increases. Based on these insights, we propose Task Switch (T-Switch), which decomposes task vectors into three components: 1) an activation switch instantiated by a binarized mask vector, 2) a polarity switch instantiated by a binarized sign vector, and 3) a scaling knob instantiated by a scalar coefficient. By storing task vectors in a binarized form, T-Switch alleviates parameter conflicts while ensuring efficient task parameter storage. Furthermore, to enable automated switch combination in T-Switch, we further introduce Auto-Switch, which enables training-free switch combination via retrieval from a small query set. Experiments indicate that our methods achieve significant performance improvements over existing baselines, requiring only 1-3% of the storage space of full-precision parameters.