Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks

TL;DR

This paper reveals that large language models struggle with fallacious reasoning, which can be exploited to bypass safety measures and generate harmful outputs through a novel jailbreak attack method.

Contribution

The paper introduces a new jailbreak technique exploiting fallacy failure in language models to produce malicious outputs, surpassing previous methods in effectiveness.

Findings

Our approach achieves more harmful outputs than previous jailbreak methods.

Language models tend to leak truthful information when asked to generate deceptive content.

Fallacious reasoning can be exploited to bypass safety mechanisms in large language models.

Abstract

We find that language models have difficulties generating fallacious and deceptive reasoning. When asked to generate deceptive outputs, language models tend to leak honest counterparts but believe them to be false. Exploiting this deficiency, we propose a jailbreak attack method that elicits an aligned language model for malicious output. Specifically, we query the model to generate a fallacious yet deceptively real procedure for the harmful behavior. Since a fallacious procedure is generally considered fake and thus harmless by LLMs, it helps bypass the safeguard mechanism. Yet the output is factually harmful since the LLM cannot fabricate fallacious solutions but proposes truthful ones. We evaluate our approach over five safety-aligned large language models, comparing four previous jailbreak methods, and show that our approach achieves competitive performance with more harmful outputs. We believe the findings could be extended beyond model safety, such as self-verification and hallucination.