InverseScope: Scalable Activation Inversion for Interpreting Large Language Models

TL;DR

InverseScope introduces a scalable, assumption-light input inversion framework for interpreting large language models' internal activations, enabling systematic analysis of their representations in practical settings.

Contribution

It proposes a novel, efficient input inversion method with a new generation architecture and evaluation protocol, advancing interpretability of large models.

Findings

Improved sample efficiency over previous methods

Scalable analysis of large language models

Quantitative evaluation of interpretability hypotheses

Abstract

Understanding the internal representations of large language models (LLMs) is a central challenge in interpretability research. Existing feature interpretability methods often rely on strong assumptions about the structure of representations that may not hold in practice. In this work, we introduce InverseScope, an assumption-light and scalable framework for interpreting neural activations via input inversion. Given a target activation, we define a distribution over inputs that generate similar activations and analyze this distribution to infer the encoded information. To address the inefficiency of sampling in high-dimensional spaces, we propose a novel conditional generation architecture that significantly improves sample efficiency compared to previous method. We further introduce a quantitative evaluation protocol that tests interpretability hypotheses using the feature consistency rate computed over the sampled inputs. InverseScope scales inversion-based interpretability methods to larger models and practical tasks, enabling systematic and quantitative analysis of internal representations in real-world LLMs.