EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents

TL;DR

EmbodiedEval is a comprehensive benchmark with 328 diverse embodied tasks in 3D environments designed to evaluate multimodal large language models' embodied capabilities, revealing significant gaps compared to human performance.

Contribution

We introduce EmbodiedEval, a new interactive benchmark with diverse tasks and scenes to evaluate MLLMs' embodied skills, addressing limitations of previous static and task-specific benchmarks.

Findings

State-of-the-art MLLMs perform significantly below humans on embodied tasks.

EmbodiedEval covers five categories: navigation, object interaction, social interaction, attribute and spatial question answering.

The benchmark reveals current MLLMs' limitations in embodied AI capabilities.

Abstract

Multimodal Large Language Models (MLLMs) have shown significant advancements, providing a promising future for embodied agents. Existing benchmarks for evaluating MLLMs primarily utilize static images or videos, limiting assessments to non-interactive scenarios. Meanwhile, existing embodied AI benchmarks are task-specific and not diverse enough, which do not adequately evaluate the embodied capabilities of MLLMs. To address this, we propose EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied 3D scenes, each of which is rigorously selected and annotated. It covers a broad spectrum of existing embodied AI tasks with significantly enhanced diversity, all within a unified simulation and evaluation framework tailored for MLLMs. The tasks are organized into five categories: navigation, object interaction, social interaction, attribute question answering, and spatial question answering to assess different capabilities of the agents. We evaluated the state-of-the-art MLLMs on EmbodiedEval and found that they have a significant shortfall compared to human level on embodied tasks. Our analysis demonstrates the limitations of existing MLLMs in embodied capabilities, providing insights for their future development. We open-source all evaluation data and simulation framework at https://github.com/thunlp/EmbodiedEval.