LONG3R: Long Sequence Streaming 3D Reconstruction

TL;DR

LONG3R introduces a real-time streaming 3D reconstruction model capable of handling long sequences by using a recurrent architecture with dynamic memory management, significantly improving performance over existing methods.

Contribution

The paper presents a novel recurrent model with a 3D spatio-temporal memory and a curriculum training strategy for efficient long-sequence 3D scene reconstruction.

Findings

Outperforms state-of-the-art streaming methods on long sequences

Maintains real-time inference speed

Effectively captures long-term scene information

Abstract

Recent advancements in multi-view scene reconstruction have been significant, yet existing methods face limitations when processing streams of input images. These methods either rely on time-consuming offline optimization or are restricted to shorter sequences, hindering their applicability in real-time scenarios. In this work, we propose LONG3R (LOng sequence streaming 3D Reconstruction), a novel model designed for streaming multi-view 3D scene reconstruction over longer sequences. Our model achieves real-time processing by operating recurrently, maintaining and updating memory with each new observation. We first employ a memory gating mechanism to filter relevant memory, which, together with a new observation, is fed into a dual-source refined decoder for coarse-to-fine interaction. To effectively capture long-sequence memory, we propose a 3D spatio-temporal memory that dynamically prunes redundant spatial information while adaptively adjusting resolution along the scene. To enhance our model's performance on long sequences while maintaining training efficiency, we employ a two-stage curriculum training strategy, each stage targeting specific capabilities. Experiments demonstrate that LONG3R outperforms state-of-the-art streaming methods, particularly for longer sequences, while maintaining real-time inference speed. Project page: https://zgchen33.github.io/LONG3R/.