Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging

TL;DR

This paper evaluates how neural scene representations like NeRFs can be adapted for multi-modal data, specifically thermal imaging, by testing four strategies and introducing a new dataset, ThermalMix.

Contribution

It systematically compares four methods for integrating a second modality into NeRFs, with a focus on thermal imaging, and provides a new multi-view dataset for multi-modal scene representation evaluation.

Findings

Adding a second branch yields the best results for thermal view synthesis.

The proposed strategies also perform well on other modalities like NIR and depth.

ThermalNeRF effectively integrates thermal data into neural scene representations.

Abstract

Neural Radiance Fields (NeRFs) quickly evolved as the new de-facto standard for the task of novel view synthesis when trained on a set of RGB images. In this paper, we conduct a comprehensive evaluation of neural scene representations, such as NeRFs, in the context of multi-modal learning. Specifically, we present four different strategies of how to incorporate a second modality, other than RGB, into NeRFs: (1) training from scratch independently on both modalities; (2) pre-training on RGB and fine-tuning on the second modality; (3) adding a second branch; and (4) adding a separate component to predict (color) values of the additional modality. We chose thermal imaging as second modality since it strongly differs from RGB in terms of radiosity, making it challenging to integrate into neural scene representations. For the evaluation of the proposed strategies, we captured a new publicly available multi-view dataset, ThermalMix, consisting of six common objects and about 360 RGB and thermal images in total. We employ cross-modality calibration prior to data capturing, leading to high-quality alignments between RGB and thermal images. Our findings reveal that adding a second branch to NeRF performs best for novel view synthesis on thermal images while also yielding compelling results on RGB. Finally, we also show that our analysis generalizes to other modalities, including near-infrared images and depth maps. Project page: https://mert-o.github.io/ThermalNeRF/.