Preemptive Answer "Attacks" on Chain-of-Thought Reasoning

TL;DR

This paper investigates how preemptive answers, obtained before reasoning, undermine the reasoning ability of large language models using Chain-of-Thought prompting, and proposes measures to improve robustness.

Contribution

It introduces the concept of preemptive answers in LLM reasoning and proposes mitigation strategies to enhance robustness against such attacks.

Findings

Preemptive answers significantly impair reasoning performance.

Preemptive answers can be induced by prompt injection attacks.

Proposed measures can partially mitigate the impact of preemptive answers.

Abstract

Large language models (LLMs) showcase impressive reasoning capabilities when coupled with Chain-of-Thought (CoT) prompting. However, the robustness of this approach warrants further investigation. In this paper, we introduce a novel scenario termed preemptive answers, where the LLM obtains an answer before engaging in reasoning. This situation can arise inadvertently or induced by malicious users by prompt injection attacks. Experiments reveal that preemptive answers significantly impair the model's reasoning capability across various CoT methods and a broad spectrum of datasets. To bolster the robustness of reasoning, we propose two measures aimed at mitigating this issue to some extent.