Fusion Steering: Prompt-Specific Activation Control

TL;DR

Fusion Steering introduces a dynamic, prompt-specific activation control method that significantly enhances factual accuracy in large language models for question-answering tasks by injecting tailored activation deltas across all transformer layers.

Contribution

This work presents a novel, flexible activation steering approach that employs prompt-specific, dynamic injection of activation deltas across the entire network, outperforming traditional fixed-layer methods.

Findings

Segmented steering achieves 25.4% accuracy on SimpleQA prompts, outperforming baseline and full-layer steering.

Boosts fully correct responses from 0.0% to 13.1% under stricter evaluation.

Demonstrates effectiveness of per-prompt, full-network activation control in improving LLM factual accuracy.

Abstract

We present Fusion Steering, an activation steering methodology that improves factual accuracy in large language models (LLMs) for question-answering (QA) tasks. This approach introduces flexible steering configurations, including full-layer steering and segmented steering. Unlike traditional methods constrained to single-layer or fixed-layer operations, Fusion Steering employs dynamic injection of prompt-specific activation deltas across all transformer layers. These activation deltas are derived from reference completions that combine the ground-truth answer with a model-generated explanation to facilitate semantically enriched, example-specific steering. The injection weights are optimized per prompt using Optuna, targeting a joint objective that balances token overlap (factual alignment) and perplexity (fluency proxy). Evaluation employs a composite score integrating token overlap and LLM-graded quality, encompassing factual accuracy, coherence, and relevance. Empirical results on 260 SimpleQA prompts (selected from 500 where the baseline failed) showcase the efficacy of segmented steering. Using Gemma-2-2B-IT with 8-bit quantization, segmented steering achieves an accuracy of 25.4% (outputs scoring $\geq 0.6$), outperforming the baseline at 3.5% and full-layer steering at 16.2%. Under the stricter SimpleQA rubric, segmented steering boosts fully correct responses from 0.0% to 13.1%. These findings highlight the strengths of segmented, dynamic intervention strategies and the promise of per-prompt, full-network activation control. Fusion Steering is also amenable to sparse representations, such as Neuronpedia or sparse crosscoders, suggesting a promising direction for interpretable and scalable activation-level control in LLMs.