SALVE: Sparse Autoencoder-Latent Vector Editing for Mechanistic Control of Neural Networks

TL;DR

SALVE introduces a novel autoencoder-based framework that enables interpretable feature discovery and precise, permanent model editing in neural networks, enhancing transparency and control.

Contribution

The paper presents SALVE, a unified framework that combines sparse autoencoder feature learning with model editing, bridging interpretability and control in neural networks.

Findings

Validated on ResNet-18 and ViT-B/16 models

Achieved interpretable control over model behavior

Provided a method for precise weight-space interventions

Abstract

Deep neural networks achieve impressive performance but remain difficult to interpret and control. We present SALVE (Sparse Autoencoder-Latent Vector Editing), a unified "discover, validate, and control" framework that bridges mechanistic interpretability and model editing. Using an $\ell_1$-regularized autoencoder, we learn a sparse, model-native feature basis without supervision. We validate these features with Grad-FAM, a feature-level saliency mapping method that visually grounds latent features in input data. Leveraging the autoencoder's structure, we perform precise and permanent weight-space interventions, enabling continuous modulation of both class-defining and cross-class features. We further derive a critical suppression threshold, $\alpha_{crit}$, quantifying each class's reliance on its dominant feature, supporting fine-grained robustness diagnostics. Our approach is validated on both convolutional (ResNet-18) and transformer-based (ViT-B/16) models, demonstrating consistent, interpretable control over their behavior. This work contributes a principled methodology for turning feature discovery into actionable model edits, advancing the development of transparent and controllable AI systems.