Geometry Meets Light: Leveraging Geometric Priors for Universal Photometric Stereo under Limited Multi-Illumination Cues

TL;DR

This paper introduces GeoUniPS, a universal photometric stereo network that leverages geometric priors from large-scale 3D models and realistic perspective projection to improve surface normal recovery in complex, in-the-wild scenes with unreliable multi-illumination cues.

Contribution

It proposes a novel network integrating geometric priors from pretrained 3D models and realistic perspective projection, enhancing photometric stereo performance in challenging scenes.

Findings

Achieves state-of-the-art results on multiple datasets.

Performs well in complex in-the-wild scenes.

Effectively handles unreliable multi-illumination cues.

Abstract

Universal Photometric Stereo is a promising approach for recovering surface normals without strict lighting assumptions. However, it struggles when multi-illumination cues are unreliable, such as under biased lighting or in shadows or self-occluded regions of complex in-the-wild scenes. We propose GeoUniPS, a universal photometric stereo network that integrates synthetic supervision with high-level geometric priors from large-scale 3D reconstruction models pretrained on massive in-the-wild data. Our key insight is that these 3D reconstruction models serve as visual-geometry foundation models, inherently encoding rich geometric knowledge of real scenes. To leverage this, we design a Light-Geometry Dual-Branch Encoder that extracts both multi-illumination cues and geometric priors from the frozen 3D reconstruction model. We also address the limitations of the conventional orthographic projection assumption by introducing the PS-Perp dataset with realistic perspective projection to enable learning of spatially varying view directions. Extensive experiments demonstrate that GeoUniPS delivers state-of-the-arts performance across multiple datasets, both quantitatively and qualitatively, especially in the complex in-the-wild scenes.