Cross-Spectral Neural Radiance Fields

TL;DR

X-NeRF introduces a method to learn unified scene representations from multi-spectral images using Neural Radiance Fields, enabling cross-spectral rendering and alignment across different camera sensitivities.

Contribution

It presents X-NeRF, a novel approach that models cross-spectral scenes with camera pose optimization and NXDC, allowing consistent rendering across spectral modalities.

Findings

Effective modeling of cross-spectral scenes demonstrated on 16 datasets.

Enables rendering of images from different spectra at arbitrary viewpoints.

Aligns multi-spectral images at the same resolution for improved analysis.

Abstract

We propose X-NeRF, a novel method to learn a Cross-Spectral scene representation given images captured from cameras with different light spectrum sensitivity, based on the Neural Radiance Fields formulation. X-NeRF optimizes camera poses across spectra during training and exploits Normalized Cross-Device Coordinates (NXDC) to render images of different modalities from arbitrary viewpoints, which are aligned and at the same resolution. Experiments on 16 forward-facing scenes, featuring color, multi-spectral and infrared images, confirm the effectiveness of X-NeRF at modeling Cross-Spectral scene representations.