Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Task

TL;DR

This paper introduces EscapeCraft-4D, a novel 4D environment to evaluate multimodal large models' ability to perform time-aware, cross-modal perception and reasoning under dynamic, time-sensitive conditions.

Contribution

It presents a new environment and benchmark for assessing temporal awareness and cross-modal integration in large models, addressing limitations of previous 2D/3D focused environments.

Findings

Models struggle with modality bias.

Significant gaps in integrating modalities under time constraints.

Insights into modality interactions in complex reasoning.

Abstract

Multimodal Large Language Models (MLLMs) have recently made rapid progress toward unified Omni models that integrate vision, language, and audio. However, existing environments largely focus on 2D or 3D visual context and vision-language tasks, offering limited support for temporally dependent auditory signals and selective cross-modal integration, where different modalities may provide complementary or interfering information, which are essential capabilities for realistic multimodal reasoning. As a result, whether models can actively coordinate modalities and reason under time-varying, irreversible conditions remains underexplored. To this end, we introduce \textbf{EscapeCraft-4D}, a customizable 4D environment for assessing selective cross-modal perception and time awareness in Omni models. It incorporates trigger-based auditory sources, temporally transient evidence, and location-dependent cues, requiring agents to perform spatio-temporal reasoning and proactive multimodal integration under time constraints. Building on this environment, we curate a benchmark to evaluate corresponding abilities across powerful models. Evaluation results suggest that models struggle with modality bias, and reveal significant gaps in current model's ability to integrate multiple modalities under time constraints. Further in-depth analysis uncovers how multiple modalities interact and jointly influence model decisions in complex multimodal reasoning environments.