Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory

TL;DR

Point3R introduces an explicit spatial pointer memory for streaming 3D reconstruction, enabling efficient and accurate integration of scene information over time, outperforming implicit memory methods.

Contribution

The paper presents Point3R, a novel online framework with explicit spatial pointer memory for dense streaming 3D reconstruction, improving capacity and accuracy over implicit memory approaches.

Findings

Achieves state-of-the-art performance on 3D reconstruction tasks.

Maintains low training costs while improving accuracy.

Effectively integrates scene information over time with explicit memory.

Abstract

Dense 3D scene reconstruction from an ordered sequence or unordered image collections is a critical step when bringing research in computer vision into practical scenarios. Following the paradigm introduced by DUSt3R, which unifies an image pair densely into a shared coordinate system, subsequent methods maintain an implicit memory to achieve dense 3D reconstruction from more images. However, such implicit memory is limited in capacity and may suffer from information loss of earlier frames. We propose Point3R, an online framework targeting dense streaming 3D reconstruction. To be specific, we maintain an explicit spatial pointer memory directly associated with the 3D structure of the current scene. Each pointer in this memory is assigned a specific 3D position and aggregates scene information nearby in the global coordinate system into a changing spatial feature. Information extracted from the latest frame interacts explicitly with this pointer memory, enabling dense integration of the current observation into the global coordinate system. We design a 3D hierarchical position embedding to promote this interaction and design a simple yet effective fusion mechanism to ensure that our pointer memory is uniform and efficient. Our method achieves competitive or state-of-the-art performance on various tasks with low training costs. Code: https://github.com/YkiWu/Point3R.