Neural Spectro-polarimetric Fields

TL;DR

This paper introduces NeSpoF, a neural model for representing the complex spatial distribution of light's spectrum and polarization, validated on a new dataset of hyperspectral-polarimetric images.

Contribution

We propose NeSpoF, the first neural representation for modeling continuous spectro-polarimetric fields, capturing physically-valid Stokes vectors across space, direction, and wavelength.

Findings

NeSpoF effectively models high-dimensional spectro-polarimetric data.

The dataset includes synthetic and real-world scenes captured with a novel imaging system.

NeSpoF demonstrates accurate representation across diverse scenes.

Abstract

Modeling the spatial radiance distribution of light rays in a scene has been extensively explored for applications, including view synthesis. Spectrum and polarization, the wave properties of light, are often neglected due to their integration into three RGB spectral bands and their non-perceptibility to human vision. However, these properties are known to encompass substantial material and geometric information about a scene. Here, we propose to model spectro-polarimetric fields, the spatial Stokes-vector distribution of any light ray at an arbitrary wavelength. We present Neural Spectro-polarimetric Fields (NeSpoF), a neural representation that models the physically-valid Stokes vector at given continuous variables of position, direction, and wavelength. NeSpoF manages inherently noisy raw measurements, showcases memory efficiency, and preserves physically vital signals - factors that are crucial for representing the high-dimensional signal of a spectro-polarimetric field. To validate NeSpoF, we introduce the first multi-view hyperspectral-polarimetric image dataset, comprised of both synthetic and real-world scenes. These were captured using our compact hyperspectral-polarimetric imaging system, which has been calibrated for robustness against system imperfections. We demonstrate the capabilities of NeSpoF on diverse scenes.