Online3R: Online Learning for Consistent Sequential Reconstruction Based on Geometry Foundation Model

TL;DR

Online3R is an online learning framework for sequential 3D reconstruction that adapts to new scenes using lightweight prompts and self-supervised strategies, improving consistency and efficiency.

Contribution

It introduces a novel online learning approach with lightweight prompts and self-supervised constraints to enhance scene reconstruction consistency in real-time.

Findings

Outperforms previous state-of-the-art methods on various benchmarks.

Effectively adapts to new environments with minimal computational overhead.

Maintains high reconstruction quality through local and global consistency constraints.

Abstract

We present Online3R, a new sequential reconstruction framework that is capable of adapting to new scenes through online learning, effectively resolving inconsistency issues. Specifically, we introduce a set of learnable lightweight visual prompts into a pretrained, frozen geometry foundation model to capture the knowledge of new environments while preserving the fundamental capability of the foundation model for geometry prediction. To solve the problems of missing groundtruth and the requirement of high efficiency when updating these visual prompts at test time, we introduce a local-global self-supervised learning strategy by enforcing the local and global consistency constraints on predictions. The local consistency constraints are conducted on intermediate and previously local fused results, enabling the model to be trained with high-quality pseudo groundtruth signals; the global consistency constraints are operated on sparse keyframes spanning long distances rather than per frame, allowing the model to learn from a consistent prediction over a long trajectory in an efficient way. Our experiments demonstrate that Online3R outperforms previous state-of-the-art methods on various benchmarks. Project page: https://shunkaizhou.github.io/online3r-1.0/