LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models

TL;DR

LAST introduces a novel approach for general vision-language models to jointly understand 3D space and long videos by enabling visual thinking trajectories, significantly improving performance in spatial, video, and image understanding tasks.

Contribution

The paper presents LAST, a method that enhances VLMs to think in space and time using only 2D images, unifying 3D and video understanding without specialized architectures.

Findings

15.8% gains on EgoSchema in zero-shot setting

8.3 gains on VSI-Bench over Qwen2.5-VL-7B

Substantial improvements across multiple benchmarks

Abstract

Humans can perceive and understand 3D space and long videos from sequential visual observations. But do vision-language models (VLMs) can? Recent work demonstrates that even state-of-the-art VLMs still struggle to understand 3D space and long videos, although they are powerful in typical vision-language tasks. Current methods often rely on specialized architectural designs to improve performance for 3D tasks and video understanding tasks separately. In contrast, we propose LAST, short for LeArn to Think in Space and Time, to jointly improve 3D spatial and long video understanding for general VLMs with only a set of 2D images as inputs. LAST makes VLMs think in space and time rather than only with text before giving the final answer, building visual thinking trajectories in 3D space and temporal dimension. We demonstrate the effectiveness of LAST in two scenarios: 1) zero-shot, where we directly prompt proprietary models; and 2) fine-tuning general VLMs with data that include thinking trajectories in 3D space and time. We show that LAST brings substantial gains in various benchmarks, including 3 spatial understanding, 4 video understanding, and 3 image understanding tasks. Notably, 15.8% gains on EgoSchema with GPT-4o in a zero-shot manner and 8.3 gains on VSI-Bench compared with Qwen2.5-VL-7B.