TrojanPraise: Jailbreak LLMs via Benign Fine-Tuning

TL;DR

TrojanPraise introduces a novel fine-tuning attack that subtly manipulates large language models to bypass moderation and jailbreak them, even using benign data that appears harmless.

Contribution

The paper presents TrojanPraise, a new attack method exploiting benign data to covertly jailbreak LLMs by shifting their attitude without altering their knowledge.

Findings

Achieves up to 95.88% success rate in attacks

Evades existing moderation models effectively

Works on both open-source and commercial LLMs

Abstract

The demand of customized large language models (LLMs) has led to commercial LLMs offering black-box fine-tuning APIs, yet this convenience introduces a critical security loophole: attackers could jailbreak the LLMs by fine-tuning them with malicious data. Though this security issue has recently been exposed, the feasibility of such attacks is questionable as malicious training dataset is believed to be detectable by moderation models such as Llama-Guard-3. In this paper, we propose TrojanPraise, a novel finetuning-based attack exploiting benign and thus filter-approved data. Basically, TrojanPraise fine-tunes the model to associate a crafted word (e.g., "bruaf") with harmless connotations, then uses this word to praise harmful concepts, subtly shifting the LLM from refusal to compliance. To explain the attack, we decouple the LLM's internal representation of a query into two dimensions of knowledge and attitude. We demonstrate that successful jailbreak requires shifting the attitude while avoiding knowledge shift, a distortion in the model's understanding of the concept. To validate this attack, we conduct experiments on five opensource LLMs and two commercial LLMs under strict black-box settings. Results show that TrojanPraise achieves a maximum attack success rate of 95.88% while evading moderation.