Pitfalls in Evaluating Interpretability Agents

TL;DR

This paper examines the challenges and pitfalls of evaluating automated interpretability systems, especially those using large language models, highlighting issues with current methods and proposing an intrinsic evaluation approach.

Contribution

The paper identifies key limitations of replication-based evaluation for interpretability agents and introduces an unsupervised intrinsic evaluation method based on component interchangeability.

Findings

Replication-based evaluation can be subjective and incomplete.

Outcome-based comparisons may obscure the research process.

LLM-based systems might reproduce findings via memorization.

Abstract

Automated interpretability systems aim to reduce the need for human labor and scale analysis to increasingly large models and diverse tasks. Recent efforts toward this goal leverage large language models (LLMs) at increasing levels of autonomy, ranging from fixed one-shot workflows to fully autonomous interpretability agents. This shift creates a corresponding need to scale evaluation approaches to keep pace with both the volume and complexity of generated explanations. We investigate this challenge in the context of automated circuit analysis -- explaining the roles of model components when performing specific tasks. To this end, we build an agentic system in which a research agent iteratively designs experiments and refines hypotheses. When evaluated against human expert explanations across six circuit analysis tasks in the literature, the system appears competitive. However, closer examination reveals several pitfalls of replication-based evaluation: human expert explanations can be subjective or incomplete, outcome-based comparisons obscure the research process, and LLM-based systems may reproduce published findings via memorization or informed guessing. To address some of these pitfalls, we propose an unsupervised intrinsic evaluation based on the functional interchangeability of model components. Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems and reveals key limitations of replication-based evaluation.