TRAN-D: 2D Gaussian Splatting-based Sparse-view Transparent Object Depth Reconstruction via Physics Simulation for Scene Update

TL;DR

TRAN-D introduces a physics-informed, Gaussian splatting-based method for accurate 3D depth reconstruction of transparent objects from sparse RGB views, effectively handling dynamic scenes and improving over existing techniques.

Contribution

The paper presents a novel 2D Gaussian splatting approach combined with physics simulation for transparent object depth reconstruction, addressing challenges of sparse views and dynamic environments.

Findings

Reduces mean absolute error by over 39% on synthetic sequences.

Achieves 48.46% accuracy with only one image, outperforming baselines using six images.

Demonstrates robustness on both synthetic and real-world data.

Abstract

Understanding the 3D geometry of transparent objects from RGB images is challenging due to their inherent physical properties, such as reflection and refraction. To address these difficulties, especially in scenarios with sparse views and dynamic environments, we introduce TRAN-D, a novel 2D Gaussian Splatting-based depth reconstruction method for transparent objects. Our key insight lies in separating transparent objects from the background, enabling focused optimization of Gaussians corresponding to the object. We mitigate artifacts with an object-aware loss that places Gaussians in obscured regions, ensuring coverage of invisible surfaces while reducing overfitting. Furthermore, we incorporate a physics-based simulation that refines the reconstruction in just a few seconds, effectively handling object removal and chain-reaction movement of remaining objects without the need for rescanning. TRAN-D is evaluated on both synthetic and real-world sequences, and it consistently demonstrated robust improvements over existing GS-based state-of-the-art methods. In comparison with baselines, TRAN-D reduces the mean absolute error by over 39% for the synthetic TRansPose sequences. Furthermore, despite being updated using only one image, TRAN-D reaches a {\delta} < 2.5 cm accuracy of 48.46%, over 1.5 times that of baselines, which uses six images. Code and more results are available at https://jeongyun0609.github.io/TRAN-D/.