Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning

TL;DR

This paper introduces InstruCoT, a method that enhances LLMs to defend against prompt injection attacks by synthesizing diverse data and instruction-level reasoning, improving security without losing utility.

Contribution

The paper presents InstruCoT, a novel approach combining data synthesis and chain-of-thought fine-tuning to effectively detect and reject malicious prompts in LLMs.

Findings

InstruCoT significantly reduces vulnerability to prompt injection attacks.

It outperforms baseline methods across multiple security dimensions.

Utility performance remains unaffected by the proposed method.

Abstract

Large language model (LLM)-integrated applications have become increasingly prevalent, yet face critical security vulnerabilities from prompt injection (PI) attacks. Defending against PI attacks faces two major issues: malicious instructions can be injected through diverse vectors, and injected instructions often lack clear semantic boundaries from the surrounding context, making them difficult to identify. To address these issues, we propose InstruCoT, a model enhancement method for PI defense that synthesizes diverse training data and employs instruction-level chain-of-thought fine-tuning, enabling LLMs to effectively identify and reject malicious instructions regardless of their source or position in the context. We evaluate InstruCoT across three critical dimensions: Behavior Deviation, Privacy Leakage, and Harmful Output. Experimental results across four LLMs demonstrate that InstruCoT significantly outperforms baselines in all dimensions while maintaining utility performance without degradation