Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents

TL;DR

This paper introduces CFG-Bench, a comprehensive benchmark for evaluating fine-grained action understanding in embodied agents, revealing current models' limitations and the benefits of supervised fine-tuning.

Contribution

We present CFG-Bench, a new benchmark with videos and questions to systematically assess fine-grained action reasoning in embodied agents, and show that fine-tuning improves performance.

Findings

Leading MLLMs struggle with detailed physical instructions.

Models have limitations in higher-order reasoning about intention.

Supervised fine-tuning significantly improves embodied benchmark performance.

Abstract

Multimodal Large Language Models (MLLMs) show promising results as decision-making engines for embodied agents operating in complex, physical environments. However, existing benchmarks often prioritize high-level planning or spatial reasoning, leaving the fine-grained action intelligence required for embodied physical interaction underexplored. To address this gap, we introduce CFG-Bench, a new benchmark designed to systematically evaluate this crucial capability. CFG-Bench consists of 1,368 curated videos paired with 19,562 question-answer pairs spanning three evaluation paradigms targeting four cognitive abilities: 1) Physical Interaction, 2) Temporal-Causal Relation, 3) Intentional Understanding, and 4) Evaluative Judgment. Together, these dimensions provide a systematic framework for assessing a model's ability to translate visual observations into actionable knowledge, moving beyond mere surface-level recognition. Our comprehensive evaluation on CFG-Bench reveals that leading MLLMs struggle to produce detailed instructions for physical interactions and exhibit profound limitations in the higher-order reasoning of intention and evaluation. Moreover, supervised fine-tuning (SFT) on our data demonstrates that teaching an MLLMs to articulate fine-grained actions directly translates to significant performance gains on established embodied benchmarks. Our analysis highlights these limitations and offers insights for developing more capable and grounded embodied agents. Project page: https://cfg-bench.github.io/