Leveraging Thermal Modality to Enhance Reconstruction in Low-Light Conditions

TL;DR

This paper introduces Thermal-NeRF, a novel method that combines thermal and visible images to improve 3D scene reconstruction in low-light conditions, overcoming noise and detail loss issues.

Contribution

Thermal-NeRF is the first to integrate thermal and visible modalities for NeRF, and it introduces a new multi-view thermal-visible dataset for enhanced low-light scene reconstruction.

Findings

Thermal-NeRF outperforms previous methods in detail preservation and noise reduction.

The combined modalities improve 3D reconstruction quality.

The new dataset supports multimodal NeRF research.

Abstract

Neural Radiance Fields (NeRF) accomplishes photo-realistic novel view synthesis by learning the implicit volumetric representation of a scene from multi-view images, which faithfully convey the colorimetric information. However, sensor noises will contaminate low-value pixel signals, and the lossy camera image signal processor will further remove near-zero intensities in extremely dark situations, deteriorating the synthesis performance. Existing approaches reconstruct low-light scenes from raw images but struggle to recover texture and boundary details in dark regions. Additionally, they are unsuitable for high-speed models relying on explicit representations. To address these issues, we present Thermal-NeRF, which takes thermal and visible raw images as inputs, considering the thermal camera is robust to the illumination variation and raw images preserve any possible clues in the dark, to accomplish visible and thermal view synthesis simultaneously. Also, the first multi-view thermal and visible dataset (MVTV) is established to support the research on multimodal NeRF. Thermal-NeRF achieves the best trade-off between detail preservation and noise smoothing and provides better synthesis performance than previous work. Finally, we demonstrate that both modalities are beneficial to each other in 3D reconstruction.