Poppy: Polarization-based Plug-and-Play Guidance for Enhancing Monocular Normal Estimation

TL;DR

Poppy is a training-free framework that enhances monocular surface normal estimation by refining predictions with polarization measurements at test time, improving accuracy on challenging surfaces.

Contribution

It introduces a novel polarization-guided refinement method that works with any frozen RGB backbone without retraining, using a differentiable rendering layer.

Findings

Reduces mean angular error by 23-26% on synthetic data

Reduces mean angular error by 6-16% on real data

Works across multiple backbone architectures and benchmarks

Abstract

Monocular surface normal estimators trained on large-scale RGB-normal data often perform poorly in the edge cases of reflective, textureless, and dark surfaces. Polarization encodes surface orientation independently of texture and albedo, offering a physics-based complement for these cases. Existing polarization methods, however, require multi-view capture or specialized training data, limiting generalization. We introduce Poppy, a training-free framework that refines normals from any frozen RGB backbone using single-shot polarization measurements at test time. Keeping backbone weights frozen, Poppy optimizes per-pixel offsets to the input RGB and output normal along with a learned reflectance decomposition. A differentiable rendering layer converts the refined normals into polarization predictions and penalizes mismatches with the observed signal. Across seven benchmarks and three backbone architectures (diffusion, flow, and feed-forward), Poppy reduces mean angular error by 23-26% on synthetic data and 6-16% on real data. These results show that guiding learned RGB-based normal estimators with polarization cues at test time refines normals on challenging surfaces without retraining.