SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos

TL;DR

SLAM3R is a real-time system that uses neural networks to produce dense 3D scene reconstructions from monocular RGB videos, bypassing traditional pose optimization.

Contribution

It introduces an end-to-end neural network approach for direct 3D pointmap regression and global scene alignment without explicit camera pose estimation.

Findings

Achieves state-of-the-art accuracy and completeness in 3D reconstruction.

Operates in real-time at over 20 frames per second.

Effectively integrates local reconstructions into a globally consistent scene.

Abstract

In this paper, we introduce SLAM3R, a novel and effective system for real-time, high-quality, dense 3D reconstruction using RGB videos. SLAM3R provides an end-to-end solution by seamlessly integrating local 3D reconstruction and global coordinate registration through feed-forward neural networks. Given an input video, the system first converts it into overlapping clips using a sliding window mechanism. Unlike traditional pose optimization-based methods, SLAM3R directly regresses 3D pointmaps from RGB images in each window and progressively aligns and deforms these local pointmaps to create a globally consistent scene reconstruction - all without explicitly solving any camera parameters. Experiments across datasets consistently show that SLAM3R achieves state-of-the-art reconstruction accuracy and completeness while maintaining real-time performance at 20+ FPS. Code available at: https://github.com/PKU-VCL-3DV/SLAM3R.