NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics

TL;DR

NeISF++ introduces a unified inverse rendering approach that effectively models both conductors and dielectrics using polarization cues, improving geometry and material reconstruction especially under strong specular reflections.

Contribution

It proposes a general pBRDF for both conductors and dielectrics and a DoLP-based geometry initialization method to handle strong specular reflections.

Findings

Outperforms existing methods in geometry and material decomposition.

Effective in relighting and real dataset applications.

Supports both conductors and dielectrics in inverse rendering.

Abstract

Recent inverse rendering methods have greatly improved shape, material, and illumination reconstruction by utilizing polarization cues. However, existing methods only support dielectrics, ignoring conductors that are found everywhere in life. Since conductors and dielectrics have different reflection properties, using previous conductor methods will lead to obvious errors. In addition, conductors are glossy, which may cause strong specular reflection and is hard to reconstruct. To solve the above issues, we propose NeISF++, an inverse rendering pipeline that supports conductors and dielectrics. The key ingredient for our proposal is a general pBRDF that describes both conductors and dielectrics. As for the strong specular reflection problem, we propose a novel geometry initialization method using DoLP images. This physical cue is invariant to intensities and thus robust to strong specular reflections. Experimental results on our synthetic and real datasets show that our method surpasses the existing polarized inverse rendering methods for geometry and material decomposition as well as downstream tasks like relighting.