Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy

TL;DR

This paper uncovers a privacy risk in merging large language models, demonstrating how malicious models can steal personal data after merging, and proposes a method to conceal such attacks.

Contribution

It introduces PhiMM, a novel privacy attack method for LLM merging, and a cloaking technique to hide malicious intent, highlighting new security concerns in model merging.

Findings

Merging phishing models increases privacy breach risks.

PII leakage increased by 3.9% after merging.

MI leakage increased by 17.4% after merging.

Abstract

Model merging is a widespread technology in large language models (LLMs) that integrates multiple task-specific LLMs into a unified one, enabling the merged model to inherit the specialized capabilities of these LLMs. Most task-specific LLMs are sourced from open-source communities and have not undergone rigorous auditing, potentially imposing risks in model merging. This paper highlights an overlooked privacy risk: \textit{an unsafe model could compromise the privacy of other LLMs involved in the model merging.} Specifically, we propose PhiMM, a privacy attack approach that trains a phishing model capable of stealing privacy using a crafted privacy phishing instruction dataset. Furthermore, we introduce a novel model cloaking method that mimics a specialized capability to conceal attack intent, luring users into merging the phishing model. Once victims merge the phishing model, the attacker can extract personally identifiable information (PII) or infer membership information (MI) by querying the merged model with the phishing instruction. Experimental results show that merging a phishing model increases the risk of privacy breaches. Compared to the results before merging, PII leakage increased by 3.9\% and MI leakage increased by 17.4\% on average. We release the code of PhiMM through a link.