Revisiting JBShield: Breaking and Rebuilding Representation-Level Jailbreak Defenses

TL;DR

This paper revisits the JBShield jailbreak defense for LLMs, proposing a new attack (JB-GCG) and a robust detection method (RTV), demonstrating the importance of multi-layer representation analysis for security.

Contribution

It introduces JB-GCG, an effective adaptive attack, and RTV, a multi-layer representation-based detection method, highlighting the limitations of single-layer concept similarity defenses.

Findings

JB-GCG achieves up to 53.4% attack success rate against JBShield.

RTV detects jailbreak prompts with 0.99 AUROC, significantly improving robustness.

Adaptive attacks remain challenging, with RTV reducing success to 7%.

Abstract

Defending large language models (LLMs) against jailbreak attacks, such as Greedy Coordinate Gradient (GCG), remains a challenge, particularly under adaptive threat models where an attacker directly targets the defense mechanism. JBShield, a recent jailbreak defense with a 0% attack success rate in some settings, detects malicious prompts via two concept signals, a toxic concept and a jailbreak concept. We design JB-GCG, which modifies GCG's objective to combine two terms: refusal-direction suppression via cosine similarity between the refusal direction and hidden-state representations, and toxic-concept regularization via JBShield's own toxic concept score. Across five configurations on Llama-3-8B, JB-GCG achieves an average ASR of 46.2%, reaching up to 53.4% in the strongest setting. We further show that our attack remains effective against JBShield-M, achieving ASR up to 30.7% across evaluated settings. The attack persists across multiple JBShield recalibrations, confirming that the vulnerability is structural rather than calibration-specific. We analyze the cosine-similarity signatures of jailbreak representations and find that they occupy a distinctive region in refusal-direction fingerprint space that neither harmless nor harmful prompts inhabit. We introduce Representation Trajectory Verification (RTV), a new defense based on Mahalanobis outlier detection over multi-layer refusal-direction fingerprints. RTV attains an AUROC of 0.99 against our attack. Finally, we design and evaluate an additional adaptive attack against RTV with full white-box knowledge of the defense; the best attack achieves only 7% ASR at 13x the computational cost. Our results show that strong non-adaptive detection does not imply robustness under adaptive threat models, and that multi-layer representation consistency is a more reliable foundation for jailbreak detection than single-layer concept similarity.