ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context

TL;DR

ASTRA-bench is a comprehensive benchmark designed to evaluate AI assistants' reasoning and planning capabilities within complex, evolving personal contexts, highlighting current limitations and guiding future improvements.

Contribution

It introduces a novel, multi-turn, context-rich benchmark with a large set of scenarios, enabling assessment of AI performance in realistic, personalized settings.

Findings

State-of-the-art models struggle with high-complexity scenarios.

Argument generation is the main bottleneck in current models.

Current agents have limited ability to ground reasoning in messy personal data.

Abstract

Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning \& Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.