ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction

TL;DR

ENACT introduces a novel benchmark for evaluating embodied cognition in vision-language models through world modeling tasks based on egocentric interactions, revealing significant gaps between models and human performance.

Contribution

This paper presents ENACT, a new benchmark for assessing embodied cognition in VLMs via world modeling tasks derived from egocentric interaction data.

Findings

Models perform better on inverse tasks than forward tasks.

Performance gap between models and humans widens with interaction horizon.

Models show anthropocentric biases like right-handed action preference.

Abstract

Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.