OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks

TL;DR

OmniEAR is a comprehensive benchmark that evaluates large language models' ability to reason about physical interactions, tool use, and multi-agent coordination in diverse embodied tasks, revealing significant performance gaps and limitations.

Contribution

The paper introduces OmniEAR, a novel benchmark for assessing embodied reasoning in language models, highlighting current limitations and architectural challenges.

Findings

Models perform well with explicit instructions (85-96%)

Performance drops significantly in tool reasoning and implicit collaboration

Complete environmental information can impair coordination performance

Abstract

Large language models excel at abstract reasoning but their capacity for embodied agent reasoning remains largely unexplored. We present OmniEAR, a comprehensive framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks. Unlike existing benchmarks that provide predefined tool sets or explicit collaboration directives, OmniEAR requires agents to dynamically acquire capabilities and autonomously determine coordination strategies based on task demands. Through text-based environment representation, we model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains. Our systematic evaluation reveals severe performance degradation when models must reason from constraints: while achieving 85-96% success with explicit instructions, performance drops to 56-85% for tool reasoning and 63-85% for implicit collaboration, with compound tasks showing over 50% failure rates. Surprisingly, complete environmental information degrades coordination performance, indicating models cannot filter task-relevant constraints. Fine-tuning improves single-agent tasks dramatically (0.6% to 76.3%) but yields minimal multi-agent gains (1.5% to 5.5%), exposing fundamental architectural limitations. These findings demonstrate that embodied reasoning poses fundamentally different challenges than current models can address, establishing OmniEAR as a rigorous benchmark for evaluating and advancing embodied AI systems. Our code and data are included in the supplementary materials and will be open-sourced upon acceptance.