Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging
Mert \"Ozer, Maximilian Weiherer, Martin Hundhausen, Bernhard Egger

TL;DR
This paper evaluates how neural scene representations like NeRFs can be adapted for multi-modal data, specifically thermal imaging, by testing four strategies and introducing a new dataset, ThermalMix.
Contribution
It systematically compares four methods for integrating a second modality into NeRFs, with a focus on thermal imaging, and provides a new multi-view dataset for multi-modal scene representation evaluation.
Findings
Adding a second branch yields the best results for thermal view synthesis.
The proposed strategies also perform well on other modalities like NIR and depth.
ThermalNeRF effectively integrates thermal data into neural scene representations.
Abstract
Neural Radiance Fields (NeRFs) quickly evolved as the new de-facto standard for the task of novel view synthesis when trained on a set of RGB images. In this paper, we conduct a comprehensive evaluation of neural scene representations, such as NeRFs, in the context of multi-modal learning. Specifically, we present four different strategies of how to incorporate a second modality, other than RGB, into NeRFs: (1) training from scratch independently on both modalities; (2) pre-training on RGB and fine-tuning on the second modality; (3) adding a second branch; and (4) adding a separate component to predict (color) values of the additional modality. We chose thermal imaging as second modality since it strongly differs from RGB in terms of radiosity, making it challenging to integrate into neural scene representations. For the evaluation of the proposed strategies, we captured a new publicly…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInfrared Target Detection Methodologies · Infrared Thermography in Medicine · Thermography and Photoacoustic Techniques
MethodsSparse Evolutionary Training
