TCLC-GS: Tightly Coupled LiDAR-Camera Gaussian Splatting for Autonomous Driving

TL;DR

This paper introduces TCLC-GS, a novel method that tightly couples LiDAR and camera data for high-quality, real-time 3D reconstruction and view synthesis in urban scenes, outperforming existing approaches.

Contribution

We propose a hybrid explicit-implicit 3D representation that fully leverages LiDAR-camera data fusion for improved 3D Gaussian splatting and reconstruction.

Findings

Achieves state-of-the-art performance on Waymo and nuScenes datasets.

Real-time RGB and depth rendering at 90-120 FPS.

Utilizes a single GPU for fast training and high-resolution outputs.

Abstract

Most 3D Gaussian Splatting (3D-GS) based methods for urban scenes initialize 3D Gaussians directly with 3D LiDAR points, which not only underutilizes LiDAR data capabilities but also overlooks the potential advantages of fusing LiDAR with camera data. In this paper, we design a novel tightly coupled LiDAR-Camera Gaussian Splatting (TCLC-GS) to fully leverage the combined strengths of both LiDAR and camera sensors, enabling rapid, high-quality 3D reconstruction and novel view RGB/depth synthesis. TCLC-GS designs a hybrid explicit (colorized 3D mesh) and implicit (hierarchical octree feature) 3D representation derived from LiDAR-camera data, to enrich the properties of 3D Gaussians for splatting. 3D Gaussian's properties are not only initialized in alignment with the 3D mesh which provides more completed 3D shape and color information, but are also endowed with broader contextual information through retrieved octree implicit features. During the Gaussian Splatting optimization process, the 3D mesh offers dense depth information as supervision, which enhances the training process by learning of a robust geometry. Comprehensive evaluations conducted on the Waymo Open Dataset and nuScenes Dataset validate our method's state-of-the-art (SOTA) performance. Utilizing a single NVIDIA RTX 3090 Ti, our method demonstrates fast training and achieves real-time RGB and depth rendering at 90 FPS in resolution of 1920x1280 (Waymo), and 120 FPS in resolution of 1600x900 (nuScenes) in urban scenarios.