PhyGaP: Physically-Grounded Gaussians with Polarization Cues

TL;DR

PhyGaP introduces a physically-grounded 3D Gaussian Splatting method utilizing polarization cues for accurate reflection decomposition, surface normal estimation, and high-fidelity relighting of reflective objects in both synthetic and real scenes.

Contribution

It leverages polarization cues with a novel PolarDR process and GridMap technique to improve 3D object reconstruction and relighting beyond RGB-based methods.

Findings

Outperforms existing RGB-based methods in PSNR and Cosine Distance.

Achieves state-of-the-art inverse rendering and relighting.

Effective even with partial polarization cues.

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) have demonstrated great success in modeling reflective 3D objects and their interaction with the environment via deferred rendering (DR). However, existing methods often struggle with correctly reconstructing physical attributes such as albedo and reflectance, and therefore they do not support high-fidelity relighting. Observing that this limitation stems from the lack of shape and material information in RGB images, we present PhyGaP, a physically-grounded 3DGS method that leverages polarization cues to facilitate precise reflection decomposition and visually consistent relighting of reconstructed objects. Specifically, we design a polarimetric deferred rendering (PolarDR) process to model polarization by reflection, and a self-occlusion-aware environment map building technique (GridMap) to resolve indirect lighting of non-convex objects. We validate on multiple synthetic and real-world scenes, including those featuring only partial polarization cues, that PhyGaP not only excels in reconstructing the appearance and surface normal of reflective 3D objects (~2 dB in PSNR and 45.7% in Cosine Distance better than existing RGB-based methods on average), but also achieves state-of-the-art inverse rendering and relighting capability. Our code will be released soon.