ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents

TL;DR

ClawForge introduces a benchmark framework for evaluating command-line agents in realistic, state-conflicted workflows, revealing diverse failure modes and limited accuracy across models.

Contribution

The paper presents ClawForge, a generator-backed, executable benchmark framework for testing agents in state-conflicted command-line workflows, enabling detailed failure analysis.

Findings

Best model achieves only 45.3% strict accuracy.

Wrong-state replacement remains below 17% for all models.

Model performance varies significantly based on state inspection behavior.

Abstract

Interactive agent benchmarks face a tension between scalable construction and realistic workflow evaluation. Hand-authored tasks are expensive to extend and revise, while static prompt evaluation misses failures that only appear when agents operate over persistent state. Existing interactive benchmarks have advanced agent evaluation significantly, but most initialize tasks from clean state and do not systematically test how agents handle pre-existing partial, stale, or conflicting artifacts. We present \textbf{ClawForge}, a generator-backed benchmark framework for executable command-line workflows under state conflict. The framework compiles scenario templates, grounded slots, initialized state, reference trajectories, and validators into reproducible task specifications, and evaluates agents step by step over persistent workflow surfaces using normalized end state and observable side effects rather than exact trajectory matching. We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories). Results across seven frontier models show that the best model reaches only 45.3% strict accuracy, wrong-state replacement remains below 17\% for all models, and the widest model separation (17% to 90%) is driven by whether agents inspect existing state before acting. Partial-credit and step-efficiency analyses further reveal that many failures are near-miss closures rather than early breakdowns, and that models exhibit qualitatively different failure styles under state conflict.