TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoning

TL;DR

TRACER is a new trajectory-level uncertainty metric for AI agents that detects critical failures in multi-turn tool-using interactions, improving early failure prediction over existing methods.

Contribution

The paper introduces TRACER, a novel uncertainty metric that combines multiple signals to identify critical episodes in agentic reasoning, addressing limitations of single-shot uncertainty proxies.

Findings

TRACER improves AUROC by up to 37.1% over baselines.

TRACER enhances AUARC by up to 55%, enabling earlier failure detection.

Effective in complex conversational tool-use scenarios.

Abstract

Estimating uncertainty for AI agents in real-world multi-turn tool-using interaction with humans is difficult because failures are often triggered by sparse critical episodes (e.g., looping, incoherent tool use, or user-agent miscoordination) even when local generation appears confident. Existing uncertainty proxies focus on single-shot text generation and therefore miss these trajectory-level breakdown signals. We introduce TRACER, a trajectory-level uncertainty metric for dual-control Tool-Agent-User interaction. TRACER combines content-aware surprisal with situational-awareness signals, semantic and lexical repetition, and tool-grounded coherence gaps, and aggregates them using a tail-focused risk functional with a MAX-composite step risk to surface decisive anomalies. We evaluate TRACER on $\tau^2$-bench by predicting task failure and selective task execution. To this end, TRACER improves AUROC by up to 37.1% and AUARC by up to 55% over baselines, enabling earlier and more accurate detection of uncertainty in complex conversational tool-use settings. Our code and benchmark are available at https://github.com/sinatayebati/agent-tracer.