CALM: Consensus-Aware Localized Merging for Multi-Task Learning

TL;DR

CALM introduces a novel merging method for multi-task learning that aligns local task-specific information with global consensus, improving performance and robustness over existing approaches.

Contribution

The paper presents CALM, a new merging technique that effectively combines models by using localized masks aligned with global task consensus, addressing interference and detail preservation issues.

Findings

CALM outperforms existing merging methods in experiments.

CALM achieves performance close to traditional multi-task learning.

CALM demonstrates robustness across various tasks.

Abstract

Model merging aims to integrate the strengths of multiple fine-tuned models into a unified model while preserving task-specific capabilities. Existing methods, represented by task arithmetic, are typically classified into global- and local-aware methods. However, global-aware methods inevitably cause parameter interference, while local-aware methods struggle to maintain the effectiveness of task-specific details in the merged model. To address these limitations, we propose a Consensus-Aware Localized Merging (CALM) method which incorporates localized information aligned with global task consensus, ensuring its effectiveness post-merging. CALM consists of three key components: (1) class-balanced entropy minimization sampling, providing a more flexible and reliable way to leverage unsupervised data; (2) an efficient-aware framework, selecting a small set of tasks for sequential merging with high scalability; (3) a consensus-aware mask optimization, aligning localized binary masks with global task consensus and merging them conflict-free. Experiments demonstrate the superiority and robustness of our CALM, significantly outperforming existing methods and achieving performance close to traditional MTL.