Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy

TL;DR

This paper introduces Instructional Segment Embedding (ISE), a novel architectural technique that embeds instruction priority into large language models, significantly enhancing safety and instruction-following capabilities against malicious prompts.

Contribution

The paper proposes ISE, an architectural embedding method inspired by BERT, to explicitly encode instruction hierarchy in LLMs, addressing a key safety vulnerability.

Findings

Up to 15.75% increase in robust accuracy on safety benchmarks

Up to 4.1% improvement in instruction-following performance

Effective differentiation of instruction types improves model safety

Abstract

Large Language Models (LLMs) are susceptible to security and safety threats, such as prompt injection, prompt extraction, and harmful requests. One major cause of these vulnerabilities is the lack of an instruction hierarchy. Modern LLM architectures treat all inputs equally, failing to distinguish between and prioritize various types of instructions, such as system messages, user prompts, and data. As a result, lower-priority user prompts may override more critical system instructions, including safety protocols. Existing approaches to achieving instruction hierarchy, such as delimiters and instruction-based training, do not address this issue at the architectural level. We introduce the Instructional Segment Embedding (ISE) technique, inspired by BERT, to modern large language models, which embeds instruction priority information directly into the model. This approach enables models to explicitly differentiate and prioritize various instruction types, significantly improving safety against malicious prompts that attempt to override priority rules. Our experiments on the Structured Query and Instruction Hierarchy benchmarks demonstrate an average robust accuracy increase of up to 15.75% and 18.68%, respectively. Furthermore, we observe an improvement in instruction-following capability of up to 4.1% evaluated on AlpacaEval. Overall, our approach offers a promising direction for enhancing the safety and effectiveness of LLM architectures.