MM3DGS SLAM: Multi-modal 3D Gaussian Splatting for SLAM Using Vision, Depth, and Inertial Measurements

TL;DR

MM3DGS introduces a novel multi-modal 3D Gaussian-based SLAM framework that integrates vision, depth, and inertial data, achieving faster rendering, better scale awareness, and improved trajectory accuracy in real-time scene mapping.

Contribution

This work is the first to utilize 3D Gaussians with unposed images and inertial data for accurate SLAM, surpassing prior neural radiance field methods in speed and accuracy.

Findings

Achieves 3x better tracking accuracy than state-of-the-art.

Improves photometric rendering quality by 5%.

Enables real-time high-resolution 3D map rendering.

Abstract

Simultaneous localization and mapping is essential for position tracking and scene understanding. 3D Gaussian-based map representations enable photorealistic reconstruction and real-time rendering of scenes using multiple posed cameras. We show for the first time that using 3D Gaussians for map representation with unposed camera images and inertial measurements can enable accurate SLAM. Our method, MM3DGS, addresses the limitations of prior neural radiance field-based representations by enabling faster rendering, scale awareness, and improved trajectory tracking. Our framework enables keyframe-based mapping and tracking utilizing loss functions that incorporate relative pose transformations from pre-integrated inertial measurements, depth estimates, and measures of photometric rendering quality. We also release a multi-modal dataset, UT-MM, collected from a mobile robot equipped with a camera and an inertial measurement unit. Experimental evaluation on several scenes from the dataset shows that MM3DGS achieves 3x improvement in tracking and 5% improvement in photometric rendering quality compared to the current 3DGS SLAM state-of-the-art, while allowing real-time rendering of a high-resolution dense 3D map. Project Webpage: https://vita-group.github.io/MM3DGS-SLAM