Circuit Fingerprints: How Answer Tokens Encode Their Geometrical Path

TL;DR

This paper demonstrates that answer tokens in transformers encode geometric directions that can be used for circuit discovery and controlled steering, revealing a unified geometric structure underlying transformer interpretability.

Contribution

It introduces the Circuit Fingerprint hypothesis, enabling circuit discovery without gradients and showing that interpretability and controllability are two aspects of the same geometric structure.

Findings

Achieves circuit discovery comparable to gradient-based methods.

Enables controlled steering with high accuracy (69.8%).

Reveals transformer circuits as fundamentally geometric structures.

Abstract

Circuit discovery and activation steering in transformers have developed as separate research threads, yet both operate on the same representational space. Are they two views of the same underlying structure? We show they follow a single geometric principle: answer tokens, processed in isolation, encode the directions that would produce them. This Circuit Fingerprint hypothesis enables circuit discovery without gradients or causal intervention -- recovering comparable structure to gradient-based methods through geometric alignment alone. We validate this on standard benchmarks (IOI, SVA, MCQA) across four model families, achieving circuit discovery performance comparable to gradient-based methods. The same directions that identify circuit components also enable controlled steering -- achieving 69.8\% emotion classification accuracy versus 53.1\% for instruction prompting while preserving factual accuracy. Beyond method development, this read-write duality reveals that transformer circuits are fundamentally geometric structures: interpretability and controllability are two facets of the same object.