Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders

TL;DR

This paper investigates the robustness of concept representations in sparse autoencoders used with large language models, revealing their vulnerability to adversarial input perturbations and highlighting the need for improved stability for reliable model monitoring.

Contribution

The paper introduces a new framework for evaluating the robustness of SAE concept representations against adversarial perturbations, emphasizing the importance of stability in interpretability tools.

Findings

Tiny adversarial perturbations can manipulate SAE interpretations

SAE concept representations are fragile without denoising

Robustness is critical for reliable model monitoring

Abstract

Sparse autoencoders (SAEs) are commonly used to interpret the internal activations of large language models (LLMs) by mapping them to human-interpretable concept representations. While existing evaluations of SAEs focus on metrics such as the reconstruction-sparsity tradeoff, human (auto-)interpretability, and feature disentanglement, they overlook a critical aspect: the robustness of concept representations to input perturbations. We argue that robustness must be a fundamental consideration for concept representations, reflecting the fidelity of concept labeling. To this end, we formulate robustness quantification as input-space optimization problems and develop a comprehensive evaluation framework featuring realistic scenarios in which adversarial perturbations are crafted to manipulate SAE representations. Empirically, we find that tiny adversarial input perturbations can effectively manipulate concept-based interpretations in most scenarios without notably affecting the base LLM's activations. Overall, our results suggest that SAE concept representations are fragile and without further denoising or postprocessing they might be ill-suited for applications in model monitoring and oversight.