Steering Conceptual Bias via Transformer Latent-Subspace Activation

TL;DR

This paper introduces a gradient-refined activation steering method to bias language model code generation towards specific programming languages, improving control and reproducibility.

Contribution

It develops G-ACT, an adaptive activation steering framework that effectively biases large language models towards desired programming languages with minimal overhead.

Findings

G-ACT increases probe classification accuracy by 15% in LLaMA-3.2 3B.

Targeted injections improve language bias even in diffuse attention models.

Per-layer probing enables practical and reproducible concept control.

Abstract

This work examines whether activating latent subspaces in language models (LLMs) can steer scientific code generation toward a specific programming language. Five causal LLMs were first evaluated on scientific coding prompts to quantify their baseline bias among four programming languages. A static neuron-attribution method, perturbing the highest activated MLP weight for a C++ or CPP token, proved brittle and exhibited limited generalization across prompt styles and model scales. To address these limitations, a gradient-refined adaptive activation steering framework (G-ACT) was developed: per-prompt activation differences are clustered into a small set of steering directions, and lightweight per-layer probes are trained and refined online to select the appropriate steering vector. In LLaMA-3.2 3B, this approach reliably biases generation towards the CPP language by increasing the average probe classification accuracy by 15% and the early layers (0-6) improving the probe classification accuracy by 61.5% compared to the standard ACT framework. For LLaMA-3.3 70B, where attention-head signals become more diffuse, targeted injections at key layers still improve language selection. Although per-layer probing introduces a modest inference overhead, it remains practical by steering only a subset of layers and enables reproducible model behavior. These results demonstrate a scalable, interpretable and efficient mechanism for concept-level control for practical agentic systems.