OpticFusion: Multi-Modal Neural Implicit 3D Reconstruction of Microstructures by Fusing White Light Interferometry and Optical Microscopy

TL;DR

OpticFusion introduces a multi-modal neural implicit approach combining White Light Interferometry and optical microscopy to achieve detailed, color-rich 3D microstructure reconstructions, addressing limitations of traditional methods.

Contribution

This work presents the first multi-modal neural implicit method for 3D microstructure reconstruction with natural color, integrating WLI and optical microscopy data.

Findings

Achieves detailed 3D reconstructions with natural color textures.

Employs a two-step data association for pose estimation.

Demonstrates effectiveness on diverse microscale samples.

Abstract

White Light Interferometry (WLI) is a precise optical tool for measuring the 3D topography of microstructures. However, conventional WLI cannot capture the natural color of a sample's surface, which is essential for many microscale research applications that require both 3D geometry and color information. Previous methods have attempted to overcome this limitation by modifying WLI hardware and analysis software, but these solutions are often costly. In this work, we address this challenge from a computer vision multi-modal reconstruction perspective for the first time. We introduce OpticFusion, a novel approach that uses an additional digital optical microscope (OM) to achieve 3D reconstruction with natural color textures using multi-view WLI and OM images. Our method employs a two-step data association process to obtain the poses of WLI and OM data. By leveraging the neural implicit representation, we fuse multi-modal data and apply color decomposition technology to extract the sample's natural color. Tested on our multi-modal dataset of various microscale samples, OpticFusion achieves detailed 3D reconstructions with color textures. Our method provides an effective tool for practical applications across numerous microscale research fields. The source code and our real-world dataset are available at https://github.com/zju3dv/OpticFusion.