SS-SfP:Neural Inverse Rendering for Self Supervised Shape from (Mixed) Polarization

TL;DR

This paper introduces SS-SfP, a self-supervised deep learning framework that estimates 3D shape from single-view polarization images by separating diffuse and specular reflections and estimating shape without ground truth normals.

Contribution

It presents the first self-supervised learning approach for Shape from Polarization under mixed polarization conditions, overcoming limitations of previous physics-based and learning-based methods.

Findings

Effective shape estimation on diverse datasets

Outperforms existing methods in accuracy

Works without ground truth surface normals

Abstract

We present a novel inverse rendering-based framework to estimate the 3D shape (per-pixel surface normals and depth) of objects and scenes from single-view polarization images, the problem popularly known as Shape from Polarization (SfP). The existing physics-based and learning-based methods for SfP perform under certain restrictions, i.e., (a) purely diffuse or purely specular reflections, which are seldom in the real surfaces, (b) availability of the ground truth surface normals for direct supervision that are hard to acquire and are limited by the scanner's resolution, and (c) known refractive index. To overcome these restrictions, we start by learning to separate the partially-polarized diffuse and specular reflection components, which we call reflectance cues, based on a modified polarization reflection model and then estimate shape under mixed polarization through an inverse-rendering based self-supervised deep learning framework called SS-SfP, guided by the polarization data and estimated reflectance cues. Furthermore, we also obtain the refractive index as a non-linear least squares solution. Through extensive quantitative and qualitative evaluation, we establish the efficacy of the proposed framework over simple single-object scenes from DeepSfP dataset and complex in-the-wild scenes from SPW dataset in an entirely self-supervised setting. To the best of our knowledge, this is the first learning-based approach to address SfP under mixed polarization in a completely self-supervised framework.