Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging

TL;DR

This paper reveals a novel attack method called Merger-as-a-Stealer that exploits model merging to extract targeted personally identifiable information from aligned large language models, highlighting security vulnerabilities.

Contribution

It introduces a two-stage attack framework demonstrating how malicious model merging can be used to steal targeted PII from large language models.

Findings

Successful extraction of targeted PII across various models

Effective attack against multiple model merging methods

Highlights need for improved model security and defenses

Abstract

Model merging has emerged as a promising approach for updating large language models (LLMs) by integrating multiple domain-specific models into a cross-domain merged model. Despite its utility and plug-and-play nature, unmonitored mergers can introduce significant security vulnerabilities, such as backdoor attacks and model merging abuse. In this paper, we identify a novel and more realistic attack surface where a malicious merger can extract targeted personally identifiable information (PII) from an aligned model with model merging. Specifically, we propose \texttt{Merger-as-a-Stealer}, a two-stage framework to achieve this attack: First, the attacker fine-tunes a malicious model to force it to respond to any PII-related queries. The attacker then uploads this malicious model to the model merging conductor and obtains the merged model. Second, the attacker inputs direct PII-related queries to the merged model to extract targeted PII. Extensive experiments demonstrate that \texttt{Merger-as-a-Stealer} successfully executes attacks against various LLMs and model merging methods across diverse settings, highlighting the effectiveness of the proposed framework. Given that this attack enables character-level extraction for targeted PII without requiring any additional knowledge from the attacker, we stress the necessity for improved model alignment and more robust defense mechanisms to mitigate such threats.