Streaming 4D Visual Geometry Transformer

TL;DR

This paper introduces a streaming visual geometry transformer that enables real-time 3D reconstruction from videos using causal attention and knowledge distillation, achieving high efficiency and spatial consistency.

Contribution

It presents a novel causal transformer architecture for online 3D reconstruction, incorporating temporal causal attention and knowledge distillation from a bidirectional model.

Findings

Enhances inference speed in online 3D reconstruction scenarios.

Maintains competitive accuracy and spatial consistency.

Supports efficient attention operators like FlashAttention.

Abstract

Perceiving and reconstructing 3D geometry from videos is a fundamental yet challenging computer vision task. To facilitate interactive and low-latency applications, we propose a streaming visual geometry transformer that shares a similar philosophy with autoregressive large language models. We explore a simple and efficient design and employ a causal transformer architecture to process the input sequence in an online manner. We use temporal causal attention and cache the historical keys and values as implicit memory to enable efficient streaming long-term 3D reconstruction. This design can handle low-latency 3D reconstruction by incrementally integrating historical information while maintaining high-quality spatial consistency. For efficient training, we propose to distill knowledge from the dense bidirectional visual geometry grounded transformer (VGGT) to our causal model. For inference, our model supports the migration of optimized efficient attention operators (e.g., FlashAttention) from large language models. Extensive experiments on various 3D geometry perception benchmarks demonstrate that our model enhances inference speed in online scenarios while maintaining competitive performance, thereby facilitating scalable and interactive 3D vision systems. Code is available at: https://github.com/wzzheng/StreamVGGT.