LiDAR-VGGT: Cross-Modal Coarse-to-Fine Fusion for Globally Consistent and Metric-Scale Dense Mapping

TL;DR

LiDAR-VGGT introduces a two-stage fusion framework combining LiDAR odometry and VGGT to produce accurate, large-scale, colored 3D maps with improved global consistency and metric accuracy in robotics applications.

Contribution

The paper presents a novel coarse-to-fine fusion pipeline that tightly couples LiDAR inertial odometry with VGGT, addressing scalability and scale distortion issues in large environments.

Findings

Outperforms existing VGGT-based and LIVO methods in dense mapping accuracy.

Produces globally consistent, colored point clouds in large-scale environments.

Demonstrates robustness across multiple datasets.

Abstract

Reconstructing large-scale colored point clouds is an important task in robotics, supporting perception, navigation, and scene understanding. Despite advances in LiDAR inertial visual odometry (LIVO), its performance remains highly sensitive to extrinsic calibration. Meanwhile, 3D vision foundation models, such as VGGT, suffer from limited scalability in large environments and inherently lack metric scale. To overcome these limitations, we propose LiDAR-VGGT, a novel framework that tightly couples LiDAR inertial odometry with the state-of-the-art VGGT model through a two-stage coarse- to-fine fusion pipeline: First, a pre-fusion module with robust initialization refinement efficiently estimates VGGT poses and point clouds with coarse metric scale within each session. Then, a post-fusion module enhances cross-modal 3D similarity transformation, using bounding-box-based regularization to reduce scale distortions caused by inconsistent FOVs between LiDAR and camera sensors. Extensive experiments across multiple datasets demonstrate that LiDAR-VGGT achieves dense, globally consistent colored point clouds and outperforms both VGGT-based methods and LIVO baselines. The implementation of our proposed novel color point cloud evaluation toolkit will be released as open source.