Parameter-Efficient Interventions for Enhanced Model Merging

TL;DR

This paper introduces IntervMerge, a novel multi-task model merging method that uses task-specific interventions and mini-interventions to reduce representation bias and improve efficiency, outperforming existing approaches.

Contribution

The paper proposes IntervMerge, a new approach that mitigates representation bias in model merging using interventions, and introduces mini-interventions for parameter-efficient updates.

Findings

IntervMerge outperforms state-of-the-art methods.

Mini-interventions reduce parameters without performance loss.

Effective mitigation of representation bias in model merging.

Abstract

Model merging combines knowledge from task-specific models into a unified multi-task model to avoid joint training on all task data. However, current methods face challenges due to representation bias, which can interfere with tasks performance. As a remedy, we propose IntervMerge, a novel approach to multi-task model merging that effectively mitigates representation bias across the model using taskspecific interventions. To further enhance its efficiency, we introduce mini-interventions, which modify only part of the representation, thereby reducing the additional parameters without compromising performance. Experimental results demonstrate that IntervMerge consistently outperforms the state-of-the-art approaches using fewer parameters.