ThermalNeRF: Thermal Radiance Fields

TL;DR

ThermalNeRF introduces a unified multispectral radiance field framework that reconstructs 3D scenes from RGB and LWIR images, enabling thermal super-resolution and occlusion removal in challenging conditions.

Contribution

It is the first to combine RGB and thermal imaging in a radiance field for 3D scene reconstruction, addressing low resolution and limited features in thermal images.

Findings

Effective scene reconstruction across visible and infrared spectra.

Capable of thermal super-resolution and occlusion removal.

Validated on real-world handheld thermal camera data.

Abstract

Thermal imaging has a variety of applications, from agricultural monitoring to building inspection to imaging under poor visibility, such as in low light, fog, and rain. However, reconstructing thermal scenes in 3D presents several challenges due to the comparatively lower resolution and limited features present in long-wave infrared (LWIR) images. To overcome these challenges, we propose a unified framework for scene reconstruction from a set of LWIR and RGB images, using a multispectral radiance field to represent a scene viewed by both visible and infrared cameras, thus leveraging information across both spectra. We calibrate the RGB and infrared cameras with respect to each other, as a preprocessing step using a simple calibration target. We demonstrate our method on real-world sets of RGB and LWIR photographs captured from a handheld thermal camera, showing the effectiveness of our method at scene representation across the visible and infrared spectra. We show that our method is capable of thermal super-resolution, as well as visually removing obstacles to reveal objects that are occluded in either the RGB or thermal channels. Please see https://yvette256.github.io/thermalnerf for video results as well as our code and dataset release.