Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions

TL;DR

This paper introduces a novel contrastive reasoning-based jailbreak technique called POATE that effectively probes language model defenses, and proposes new reasoning methods to improve model robustness against such attacks.

Contribution

The paper presents POATE, a new contrastive reasoning attack method, and introduces Intent-Aware CoT and Reverse Thinking CoT to defend against these sophisticated jailbreaks.

Findings

POATE achieves ~44% attack success rate across six model families.

Proposed defenses significantly reduce vulnerability to contrastive reasoning attacks.

Enhanced reasoning methods improve model robustness against subtle adversarial prompts.

Abstract

Large language models, despite extensive alignment with human values and ethical principles, remain vulnerable to sophisticated jailbreak attacks that exploit their reasoning abilities. Existing safety measures often detect overt malicious intent but fail to address subtle, reasoning-driven vulnerabilities. In this work, we introduce POATE (Polar Opposite query generation, Adversarial Template construction, and Elaboration), a novel jailbreak technique that harnesses contrastive reasoning to provoke unethical responses. POATE crafts semantically opposing intents and integrates them with adversarial templates, steering models toward harmful outputs with remarkable subtlety. We conduct extensive evaluation across six diverse language model families of varying parameter sizes to demonstrate the robustness of the attack, achieving significantly higher attack success rates (~44%) compared to existing methods. To counter this, we propose Intent-Aware CoT and Reverse Thinking CoT, which decompose queries to detect malicious intent and reason in reverse to evaluate and reject harmful responses. These methods enhance reasoning robustness and strengthen the model's defense against adversarial exploits.