STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer

TL;DR

STream3R introduces a scalable, Transformer-based framework for real-time 3D reconstruction from image sequences, outperforming prior methods especially in dynamic scenes and enabling large-scale pretraining.

Contribution

It reformulates 3D pointmap prediction as a causal Transformer problem, enabling efficient streaming processing and better generalization in dynamic environments.

Findings

Outperforms prior methods on static and dynamic scene benchmarks.

Efficiently processes image sequences using causal attention.

Compatible with large-scale pretraining and fine-tuning.

Abstract

We present STream3R, a novel approach to 3D reconstruction that reformulates pointmap prediction as a decoder-only Transformer problem. Existing state-of-the-art methods for multi-view reconstruction either depend on expensive global optimization or rely on simplistic memory mechanisms that scale poorly with sequence length. In contrast, STream3R introduces an streaming framework that processes image sequences efficiently using causal attention, inspired by advances in modern language modeling. By learning geometric priors from large-scale 3D datasets, STream3R generalizes well to diverse and challenging scenarios, including dynamic scenes where traditional methods often fail. Extensive experiments show that our method consistently outperforms prior work across both static and dynamic scene benchmarks. Moreover, STream3R is inherently compatible with LLM-style training infrastructure, enabling efficient large-scale pretraining and fine-tuning for various downstream 3D tasks. Our results underscore the potential of causal Transformer models for online 3D perception, paving the way for real-time 3D understanding in streaming environments. More details can be found in our project page: https://nirvanalan.github.io/projects/stream3r.