Selective Safety Steering via Value-Filtered Decoding

TL;DR

This paper introduces a value-filtered decoding method for large language models that reduces unnecessary safety interventions while enhancing safety, balancing safety and output quality effectively.

Contribution

It proposes a novel test-time steering technique that filters tokens based on safety value, with explicit control over false intervention rates, outperforming existing methods.

Findings

Outperforms baseline safety steering methods across multiple datasets.

Provides explicit control over false intervention probability.

Improves safety without significantly compromising helpfulness or coherence.

Abstract

While large language models (LLMs) are trained to align with human values, their generations may still violate safety constraints. A growing line of work addresses this problem by modifying the model's sampling policy at decoding time using a safety reward. However, existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model. Such unnecessary interventions are undesirable, as they can distort key properties of the base model such as helpfulness, fluency, style, and coherence. We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses. Our approach filters tokens using a value-based safety criterion and provides an explicit bound on the probability of false interventions. A single threshold hyperparameter controls this bound, allowing practitioners to trade off higher rates of unnecessary intervention for better output safety. Across multiple datasets and experiments, we show that our value-filtered decoding method outperforms existing baselines, achieving better trade-offs between safety, helpfulness, and similarity to the base model.