Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Models

TL;DR

This paper introduces Merge Hijacking, a novel backdoor attack targeting the model merging process in large language models, demonstrating its effectiveness and robustness against defenses across various models and merging techniques.

Contribution

It presents the first backdoor attack specifically designed for model merging in LLMs, highlighting vulnerabilities and potential security risks.

Findings

Attack remains effective across different models and merging algorithms.

The backdoor persists even with real-world models and defenses.

The method demonstrates high effectiveness and robustness in experiments.

Abstract

Model merging for Large Language Models (LLMs) directly fuses the parameters of different models finetuned on various tasks, creating a unified model for multi-domain tasks. However, due to potential vulnerabilities in models available on open-source platforms, model merging is susceptible to backdoor attacks. In this paper, we propose Merge Hijacking, the first backdoor attack targeting model merging in LLMs. The attacker constructs a malicious upload model and releases it. Once a victim user merges it with any other models, the resulting merged model inherits the backdoor while maintaining utility across tasks. Merge Hijacking defines two main objectives-effectiveness and utility-and achieves them through four steps. Extensive experiments demonstrate the effectiveness of our attack across different models, merging algorithms, and tasks. Additionally, we show that the attack remains effective even when merging real-world models. Moreover, our attack demonstrates robustness against two inference-time defenses (Paraphrasing and CLEANGEN) and one training-time defense (Fine-pruning).