Beyond the Black Box: Interpretability of Agentic AI Tool Use

TL;DR

This paper presents a mechanistic interpretability toolkit using Sparse Autoencoders and linear probes to diagnose and understand internal decision-making processes of AI agents during tool use, enhancing safety and reliability.

Contribution

It introduces a novel framework that infers internal states related to tool decisions in agentic AI, providing deeper insight into failures and enabling better internal observability.

Findings

The toolkit successfully identifies features associated with tool decisions.

Ablation of identified features impacts the model's tool use behavior.

Applied to multiple models, it reveals internal signals preceding tool calls.

Abstract

AI agents are promising for high-stakes enterprise workflows, but dependable deployment remains limited because tool-use failures are difficult to diagnose and control. Agents may skip required tool calls, invoke tools unnecessarily, or take actions whose consequence becomes visible only after execution. Existing observability methods are mostly external: prompts reveal correlations, evaluations score outputs, and logs arrive only after the model has already acted. In long-horizon settings, these failures are especially costly because an early tool mistake can alter the rest of the trajectory, increase token consumption, and create downstream safety and security risk. We introduce a mechanistic-interpretability toolkit built on Sparse Autoencoders (SAEs) and linear probes. The framework reads model states before each action and infers both whether a tool is needed and how consequential the next tool action is likely to be. By decomposing activations into sparse features, it identifies the internal layers and features most associated with tool decisions and tests their functional importance through feature ablation. We train the probes on multi-step trajectories from the NVIDIA Nemotron function-calling dataset and apply the same workflow to GPT-OSS 20B and Gemma 3 27B models. The goal is not to replace external evaluation, but to add a missing layer: visibility into what the model signaled internally before action. This helps surface deeper causes of agent failure, especially in long-horizon runs where an early mistake can reshape the rest of the agentic interaction. More broadly, the paper shows how mechanistic interpretability can support practical internal observability for monitoring tool calls and risk in agent systems.