MM-Gaussian: 3D Gaussian-based Multi-modal Fusion for Localization and Reconstruction in Unbounded Scenes

TL;DR

MM-Gaussian introduces a multi-modal fusion system leveraging 3D Gaussians for accurate localization and realistic scene reconstruction in unbounded outdoor environments, combining LiDAR and camera data for improved robustness.

Contribution

The paper presents a novel LiDAR-camera fusion approach using 3D Gaussians for enhanced localization and rendering in unbounded scenes, with a relocalization module for robustness.

Findings

Effective localization in unbounded outdoor scenes

High-quality, realistic scene rendering

Robust relocalization capability

Abstract

Localization and mapping are critical tasks for various applications such as autonomous vehicles and robotics. The challenges posed by outdoor environments present particular complexities due to their unbounded characteristics. In this work, we present MM-Gaussian, a LiDAR-camera multi-modal fusion system for localization and mapping in unbounded scenes. Our approach is inspired by the recently developed 3D Gaussians, which demonstrate remarkable capabilities in achieving high rendering quality and fast rendering speed. Specifically, our system fully utilizes the geometric structure information provided by solid-state LiDAR to address the problem of inaccurate depth encountered when relying solely on visual solutions in unbounded, outdoor scenarios. Additionally, we utilize 3D Gaussian point clouds, with the assistance of pixel-level gradient descent, to fully exploit the color information in photos, thereby achieving realistic rendering effects. To further bolster the robustness of our system, we designed a relocalization module, which assists in returning to the correct trajectory in the event of a localization failure. Experiments conducted in multiple scenarios demonstrate the effectiveness of our method.