Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

TL;DR

Claw-Eval is a comprehensive evaluation suite for autonomous agents that addresses limitations of existing benchmarks by providing trajectory-aware grading, safety, robustness, and multimodal coverage, revealing nuanced insights into model capabilities.

Contribution

Introduces Claw-Eval, an end-to-end, multi-modal evaluation framework with detailed grading and safety metrics, improving reliability over existing benchmarks.

Findings

Trajectory-opaque evaluation misses safety and robustness issues.

Capability does not guarantee consistency across tasks.

Model rankings vary significantly across different metrics and task groups.

Abstract

Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable.