Findings of the BlackboxNLP 2025 Shared Task: Localizing Circuits and Causal Variables in Language Models

TL;DR

This paper reports on the BlackboxNLP 2025 Shared Task, which advances the evaluation of mechanistic interpretability in language models through community-wide benchmarking of circuit and causal variable localization methods.

Contribution

It extends the Mechanistic Interpretability Benchmark into a shared task, providing a standardized, reproducible framework for comparing diverse MI techniques across multiple teams.

Findings

Ensemble and regularization strategies improved circuit localization.

Low-dimensional, non-linear projections enhanced causal variable localization.

Multiple methods showed significant progress in localizing influential model components.

Abstract

Mechanistic interpretability (MI) seeks to uncover how language models (LMs) implement specific behaviors, yet measuring progress in MI remains challenging. The recently released Mechanistic Interpretability Benchmark (MIB; Mueller et al., 2025) provides a standardized framework for evaluating circuit and causal variable localization. Building on this foundation, the BlackboxNLP 2025 Shared Task extends MIB into a community-wide reproducible comparison of MI techniques. The shared task features two tracks: circuit localization, which assesses methods that identify causally influential components and interactions driving model behavior, and causal variable localization, which evaluates approaches that map activations into interpretable features. With three teams spanning eight different methods, participants achieved notable gains in circuit localization using ensemble and regularization strategies for circuit discovery. With one team spanning two methods, participants achieved significant gains in causal variable localization using low-dimensional and non-linear projections to featurize activation vectors. The MIB leaderboard remains open; we encourage continued work in this standard evaluation framework to measure progress in MI research going forward.