RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting

TL;DR

RTG-SLAM is a real-time 3D reconstruction system using Gaussian splatting that efficiently handles large-scale environments with improved speed, memory efficiency, and high-quality results compared to existing methods.

Contribution

The paper introduces a novel Gaussian-based SLAM system that reduces computational cost and memory usage while maintaining high reconstruction quality in real-time large-scale environments.

Findings

Achieves real-time large-scale 3D reconstruction with high quality.

Runs approximately twice as fast as NeRF-based methods.

Uses half the memory of comparable systems.

Abstract

We present Real-time Gaussian SLAM (RTG-SLAM), a real-time 3D reconstruction system with an RGBD camera for large-scale environments using Gaussian splatting. The system features a compact Gaussian representation and a highly efficient on-the-fly Gaussian optimization scheme. We force each Gaussian to be either opaque or nearly transparent, with the opaque ones fitting the surface and dominant colors, and transparent ones fitting residual colors. By rendering depth in a different way from color rendering, we let a single opaque Gaussian well fit a local surface region without the need of multiple overlapping Gaussians, hence largely reducing the memory and computation cost. For on-the-fly Gaussian optimization, we explicitly add Gaussians for three types of pixels per frame: newly observed, with large color errors, and with large depth errors. We also categorize all Gaussians into stable and unstable ones, where the stable Gaussians are expected to well fit previously observed RGBD images and otherwise unstable. We only optimize the unstable Gaussians and only render the pixels occupied by unstable Gaussians. In this way, both the number of Gaussians to be optimized and pixels to be rendered are largely reduced, and the optimization can be done in real time. We show real-time reconstructions of a variety of large scenes. Compared with the state-of-the-art NeRF-based RGBD SLAM, our system achieves comparable high-quality reconstruction but with around twice the speed and half the memory cost, and shows superior performance in the realism of novel view synthesis and camera tracking accuracy.