Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understanding

TL;DR

This paper introduces Motion-MLLM, a framework that enhances multimodal large language models with egomotion data for more efficient and accurate 3D scene understanding by grounding visual content in physical trajectories.

Contribution

It proposes a novel egomotion-aware framework with keyframe filtering and cross-modal fusion, improving 3D reasoning and spatial understanding in videos.

Findings

Achieves competitive accuracy in 3D scene understanding tasks.

Runs 1.30x faster than video frame-based methods.

Runs 1.61x faster than explicit 3D data methods.

Abstract

Recent Multimodal Large Language Models (MLLMs) have shown high potential for spatial reasoning within 3D scenes. However, they typically rely on computationally expensive 3D representations like point clouds or reconstructed Bird's-Eye View (BEV) maps, or lack physical grounding to resolve ambiguities in scale and size. This paper significantly enhances MLLMs with egomotion modality data, captured by Inertial Measurement Units (IMUs) concurrently with the video. In particular, we propose a novel framework, called Motion-MLLM, introducing two key components: (1) a cascaded motion-visual keyframe filtering module that leverages both IMU data and visual features to efficiently select a sparse yet representative set of keyframes, and (2) an asymmetric cross-modal fusion module where motion tokens serve as intermediaries that channel egomotion cues and cross-frame visual context into the visual representation. By grounding visual content in physical egomotion trajectories, Motion-MLLM can reason about absolute scale and spatial relationships across the scene. Our extensive evaluation shows that Motion-MLLM makes significant improvements in various tasks related to 3D scene understanding and spatial reasoning. Compared to state-of-the-art (SOTA) methods based on video frames and explicit 3D data, Motion-MLLM achieves competitive accuracy while running $1.30\times$ and $1.61\times$ faster, respectively.