Compositional Steering of Large Language Models with Steering Tokens

TL;DR

This paper introduces compositional steering tokens that enable simultaneous multi-behavior control of large language models, improving zero-shot composition and combining effectively with natural language instructions.

Contribution

It proposes a novel method for multi-behavior steering using dedicated tokens embedded via self-distillation, enabling better zero-shot compositional control of LLMs.

Findings

Steering tokens outperform existing methods in multi-behavior control.

The approach generalizes well to unseen behavior combinations.

Combining steering tokens with natural language instructions yields further improvements.

Abstract

Deploying LLMs in real-world applications requires controllable output that satisfies multiple desiderata at the same time. While existing work extensively addresses LLM steering for a single behavior, \textit{compositional steering} -- i.e., steering LLMs simultaneously towards multiple behaviors -- remains an underexplored problem. In this work, we propose \emph{compositional steering tokens} for multi-behavior steering. We first embed individual behaviors, expressed as natural language instructions, into dedicated tokens via self-distillation. Contrary to most prior work, which operates in the activation space, our behavior steers live in the space of input tokens, enabling more effective zero-shot composition. We then train a dedicated \textit{composition token} on pairs of behaviors and show that it successfully captures the notion of composition: it generalizes well to \textit{unseen} compositions, including those with unseen behaviors as well as those with an unseen \textit{number} of behaviors. Our experiments across different LLM architectures show that steering tokens lead to superior multi-behavior steering of verifiable constraints (e.g., length, format, structure, language) compared to competing approaches (instructions, activation steering, and LoRA merging). Moreover, we show that steering tokens complement natural language instructions, with their combination resulting in further gains.