Learning Safety Constraints for Large Language Models

TL;DR

SaP introduces a geometric method to enforce safety constraints in large language models by learning a safety polytope in the representation space, enabling detection and correction of unsafe outputs without altering the model weights.

Contribution

The paper presents SaP, a novel post-hoc safety framework that learns a safety polytope in the representation space to improve LLM safety and interpretability.

Findings

Effectively detects unethical inputs

Reduces adversarial attack success rates

Maintains performance on standard tasks

Abstract

Large language models (LLMs) have emerged as powerful tools but pose significant safety risks through harmful outputs and vulnerability to adversarial attacks. We propose SaP, short for Safety Polytope, a geometric approach to LLM safety that learns and enforces multiple safety constraints directly in the model's representation space. We develop a framework that identifies safe and unsafe regions via the polytope's facets, enabling both detection and correction of unsafe outputs through geometric steering. Unlike existing approaches that modify model weights, SaP operates post-hoc in the representation space, preserving model capabilities while enforcing safety constraints. Experiments across multiple LLMs demonstrate that our method can effectively detect unethical inputs, reduce adversarial attack success rates while maintaining performance on standard tasks, thus highlighting the importance of having an explicit geometric model for safety. Analysis of the learned polytope facets reveals emergence of specialization in detecting different semantic notions of safety, providing interpretable insights into how safety is captured in LLMs' representation space.