Prism-$\Delta$: Differential Subspace Steering for Prompt Highlighting in Large Language Models

TL;DR

PRISM-Δ introduces a novel differential subspace steering method for large language models that improves prompt highlighting by effectively capturing discriminative signals, leading to better performance with lower fluency costs.

Contribution

It proposes PRISM-Δ, a new framework that decomposes cross-covariance matrices to enhance prompt steering, outperforming existing methods across multiple benchmarks and models.

Findings

Matches or exceeds existing methods on 19 of 20 configurations.

Achieves up to +10.6% relative gain in performance.

Halves the fluency cost of steering.

Abstract

Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-$\Delta$ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, PRISM-$\Delta$ matches or exceeds the best existing method on 19 of 20 configurations, with relative gains up to +10.6%, while halving the fluency cost of steering. PRISM-$\Delta$ also scales to long-context retrieval, outperforming the best existing method by up to +4.8% relative gain. PRISM-$\Delta$ is compatible with FlashAttention and adds negligible memory overhead.