NeuronScope: A Multi-Agent Framework for Explaining Polysemantic Neurons in Language Models

TL;DR

NeuronScope introduces a multi-agent, iterative framework for interpreting polysemantic neurons in language models, effectively decomposing neuron activations into semantic components and outperforming traditional single-pass methods.

Contribution

It presents a novel multi-agent approach that enhances neuron interpretation by iteratively refining explanations based on activation feedback, addressing polysemanticity in LLMs.

Findings

Uncovers hidden polysemantic neurons in language models.

Produces explanations with higher activation correlation.

Outperforms single-pass interpretation baselines.

Abstract

Neuron-level interpretation in large language models (LLMs) is fundamentally challenged by widespread polysemanticity, where individual neurons respond to multiple distinct semantic concepts. Existing single-pass interpretation methods struggle to faithfully capture such multi-concept behavior. In this work, we propose NeuronScope, a multi-agent framework that reformulates neuron interpretation as an iterative, activation-guided process. NeuronScope explicitly deconstructs neuron activations into atomic semantic components, clusters them into distinct semantic modes, and iteratively refines each explanation using neuron activation feedback. Experiments demonstrate that NeuronScope uncovers hidden polysemanticity and produces explanations with significantly higher activation correlation compared to single-pass baselines.