Material-informed Gaussian Splatting for 3D World Reconstruction in a Digital Twin

TL;DR

This paper introduces a camera-only 3D scene reconstruction method using Gaussian Splatting and semantic material masks, achieving photorealistic results with physics-based material properties for digital twin applications.

Contribution

It presents a novel camera-only pipeline that reconstructs scenes with Gaussian Splatting, extracts semantic materials, and assigns physics-based properties, eliminating the need for complex LiDAR-camera calibration.

Findings

Achieves sensor simulation fidelity comparable to LiDAR-camera fusion.

Eliminates hardware complexity and calibration requirements.

Validates method using real-world dataset with LiDAR as ground truth.

Abstract

3D reconstruction for Digital Twins often relies on LiDAR-based methods, which provide accurate geometry but lack the semantics and textures naturally captured by cameras. Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass, which are visible in images but poorly represented in point clouds. We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images, extracts semantic material masks via vision models, converts Gaussian representations to mesh surfaces with projected material labels, and assigns physics-based material properties for accurate sensor simulation in modern graphics engines and simulators. This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion while eliminating hardware complexity and calibration requirements. We validate our camera-only method using an internal dataset from an instrumented test vehicle, leveraging LiDAR as ground truth for reflectivity validation alongside image similarity metrics.