Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects

TL;DR

This paper presents a novel differentiable refraction-tracing technique for high-precision 3D mesh reconstruction of transparent objects, overcoming limitations of existing methods by directly optimizing mesh geometry with silhouette and refraction constraints.

Contribution

It introduces a new method combining static background patterns and differentiable refraction tracing to accurately reconstruct transparent objects without over-smoothing.

Findings

Achieves higher reconstruction accuracy than state-of-the-art methods.

Effectively preserves fine details in transparent object meshes.

Demonstrates robustness across various transparent object shapes.

Abstract

Capturing the 3D geometry of transparent objects is a challenging task, ill-suited for general-purpose scanning and reconstruction techniques, since these cannot handle specular light transport phenomena. Existing state-of-the-art methods, designed specifically for this task, either involve a complex setup to reconstruct complete refractive ray paths, or leverage a data-driven approach based on synthetic training data. In either case, the reconstructed 3D models suffer from over-smoothing and loss of fine detail. This paper introduces a novel, high precision, 3D acquisition and reconstruction method for solid transparent objects. Using a static background with a coded pattern, we establish a mapping between the camera view rays and locations on the background. Differentiable tracing of refractive ray paths is then used to directly optimize a 3D mesh approximation of the object, while simultaneously ensuring silhouette consistency and smoothness. Extensive experiments and comparisons demonstrate the superior accuracy of our method.