Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers

TL;DR

This paper introduces Paper Summary Attack (PSA), a novel method exploiting LLMs' trust in authoritative sources by synthesizing adversarial prompts from safety papers, revealing significant vulnerabilities in current models.

Contribution

The paper presents PSA, a new attack technique that leverages safety papers to systematically generate adversarial prompts, exposing vulnerabilities in both base and aligned LLMs.

Findings

High attack success rates of 97-98% on various models.

Vulnerabilities vary across models and versions, showing bias.

PSA exposes significant security risks in current LLM safety measures.

Abstract

The safety of large language models (LLMs) has garnered significant research attention. In this paper, we argue that previous empirical studies demonstrate LLMs exhibit a propensity to trust information from authoritative sources, such as academic papers, implying new possible vulnerabilities. To verify this possibility, a preliminary analysis is designed to illustrate our two findings. Based on this insight, a novel jailbreaking method, Paper Summary Attack (\llmname{PSA}), is proposed. It systematically synthesizes content from either attack-focused or defense-focused LLM safety paper to construct an adversarial prompt template, while strategically infilling harmful query as adversarial payloads within predefined subsections. Extensive experiments show significant vulnerabilities not only in base LLMs, but also in state-of-the-art reasoning model like Deepseek-R1. PSA achieves a 97\% attack success rate (ASR) on well-aligned models like Claude3.5-Sonnet and an even higher 98\% ASR on Deepseek-R1. More intriguingly, our work has further revealed diametrically opposed vulnerability bias across different base models, and even between different versions of the same model, when exposed to either attack-focused or defense-focused papers. This phenomenon potentially indicates future research clues for both adversarial methodologies and safety alignment.Code is available at https://github.com/233liang/Paper-Summary-Attack