Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders

TL;DR

This paper introduces a Sparse Autoencoder framework for retrieving and steering high-level semantic features in LLMs, enabling precise control over linguistic behaviors with improved stability.

Contribution

It presents a novel contrastive retrieval method for internal features, demonstrating bidirectional steering of model behavior and the concept of Functional Faithfulness.

Findings

Enables precise, bidirectional control of linguistic traits in LLMs.

Introduces the concept of Functional Faithfulness for predictable feature interventions.

Outperforms existing activation steering methods like CAA in stability and performance.

Abstract

Recent work in Mechanistic Interpretability (MI) has enabled the identification and intervention of internal features in Large Language Models (LLMs). However, a persistent challenge lies in linking such internal features to the reliable control of complex, behavior-level semantic attributes in language generation. In this paper, we propose a Sparse Autoencoder-based framework for retrieving and steering semantically interpretable internal features associated with high-level linguistic behaviors. Our method employs a contrastive feature retrieval pipeline based on controlled semantic oppositions, combing statistical activation analysis and generation-based validation to distill monosemantic functional features from sparse activation spaces. Using the Big Five personality traits as a case study, we demonstrate that our method enables precise, bidirectional steering of model behavior while maintaining superior stability and performance compared to existing activation steering methods like Contrastive Activation Addition (CAA). We further identify an empirical effect, which we term Functional Faithfulness, whereby intervening on a specific internal feature induces coherent and predictable shifts across multiple linguistic dimensions aligned with the target semantic attribute. Our findings suggest that LLMs internalize deeply integrated representations of high-order concepts, and provide a novel, robust mechanistic path for the regulation of complex AI behaviors.