Improved Representation Steering for Language Models

TL;DR

This paper introduces RePS, a new method for steering language models that improves control over concepts and suppression, outperforming existing methods and offering interpretability and robustness.

Contribution

The paper proposes Reference-free Preference Steering (RePS), a novel bidirectional optimization approach for concept steering and suppression in language models, reducing reliance on prompting.

Findings

RePS outperforms existing steering methods on the AxBench benchmark.

RePS narrows the gap between representation steering and prompting.

RePS remains resilient to prompt-based jailbreaking attacks.

Abstract

Steering methods for language models (LMs) seek to provide fine-grained and interpretable control over model generations by variously changing model inputs, weights, or representations to adjust behavior. Recent work has shown that adjusting weights or representations is often less effective than steering by prompting, for instance when wanting to introduce or suppress a particular concept. We demonstrate how to improve representation steering via our new Reference-free Preference Steering (RePS), a bidirectional preference-optimization objective that jointly does concept steering and suppression. We train three parameterizations of RePS and evaluate them on AxBench, a large-scale model steering benchmark. On Gemma models with sizes ranging from 2B to 27B, RePS outperforms all existing steering methods trained with a language modeling objective and substantially narrows the gap with prompting -- while promoting interpretability and minimizing parameter count. In suppression, RePS matches the language-modeling objective on Gemma-2 and outperforms it on the larger Gemma-3 variants while remaining resilient to prompt-based jailbreaking attacks that defeat prompting. Overall, our results suggest that RePS provides an interpretable and robust alternative to prompting for both steering and suppression.