Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents

TL;DR

The paper introduces VIGIL, a new evaluation framework for embodied agents that separately measures world completion and terminal commitment, enabling clearer assessment of task success and failure modes.

Contribution

VIGIL provides a protocol to independently evaluate terminal commitment and world completion, distinguishing different failure types in embodied agent benchmarks.

Findings

VIGIL yields two scores: world-state completion and benchmark success.

Models with similar world completion can differ significantly in success due to commitment issues.

Action-feedback improves world completion but does not fully resolve commitment failures.

Abstract

Standard embodied evaluations do not independently score whether an agent correctly commits to task completion at episode closure, a capacity we call terminal commitment. Behaviorally distinct failures--never completing the task, completing it but failing to stop, and reporting success without sufficient evidence--collapse into the same benchmark failure. We introduce VIGIL, an evaluation framework that makes terminal commitment independently measurable. Under VIGIL's default protocol, agents observe only egocentric RGB, receive no action-success signals, and must end each episode with a semantic report checked deterministically against hidden world state. This yields two separate scores: world-state completion (W) and benchmark success (B), where B additionally requires a correct terminal report. This decoupling makes four outcome categories distinguishable: missed execution, post-attainment drift, unsupported commitment, and verified success. Across 20 models on 1,000 frozen episodes, systems with comparable W differ by up to 19.7 pp in B: one model converts achieved states into correct reports, while another with near-identical execution drifts past the goal without closing. An action-feedback intervention further tests the separation: execution-oriented signals improve W broadly, yet commitment failures persist in models that do not already ground terminal reports in the achieved state. VIGIL provides a protocol that makes terminal commitment independently visible and scorable.