$M^3-Verse$: A "Spot the Difference" Challenge for Large Multimodal Models

TL;DR

This paper introduces $M^3-Verse$, a comprehensive benchmark for evaluating large multimodal models' ability to understand dynamic object changes across videos, revealing current limitations and proposing a baseline for improvement.

Contribution

The paper presents a new benchmark, $M^3-Verse$, for assessing models' understanding of object transformations in videos, and evaluates existing models, highlighting their shortcomings and proposing a simple baseline.

Findings

Existing LMMs struggle with tracking state transitions.

$M^3-Verse$ contains 270 scenes and 2,932 questions.

A baseline improves multi-state perception performance.

Abstract

Modern Large Multimodal Models (LMMs) have demonstrated extraordinary ability in static image and single-state spatial-temporal understanding. However, their capacity to comprehend the dynamic changes of objects within a shared spatial context between two distinct video observations, remains largely unexplored. This ability to reason about transformations within a consistent environment is particularly crucial for advancements in the field of spatial intelligence. In this paper, we introduce $M^3-Verse$, a Multi-Modal, Multi-State, Multi-Dimensional benchmark, to formally evaluate this capability. It is built upon paired videos that provide multi-perspective observations of an indoor scene before and after a state change. The benchmark contains a total of 270 scenes and 2,932 questions, which are categorized into over 50 subtasks that probe 4 core capabilities. We evaluate 16 state-of-the-art LMMs and observe their limitations in tracking state transitions. To address these challenges, we further propose a simple yet effective baseline that achieves significant performance improvements in multi-state perception. $M^3-Verse$ thus provides a challenging new testbed to catalyze the development of next-generation models with a more holistic understanding of our dynamic visual world. You can get the construction pipeline from https://github.com/Wal-K-aWay/M3-Verse_pipeline and full benchmark data from https://www.modelscope.cn/datasets/WalKaWay/M3-Verse.