Lighting, Reflectance and Geometry Estimation from 360$^{\circ}$ Panoramic Stereo

TL;DR

This paper introduces a deep learning approach that jointly estimates high-resolution lighting, reflectance, and geometry from 360-degree stereo images, leveraging full scene observation for improved accuracy and AR applications.

Contribution

The method uniquely combines 360-degree stereo input with physical constraints to jointly estimate scene lighting, reflectance, and geometry, advancing scene understanding techniques.

Findings

Outperforms prior state-of-the-art methods in scene property estimation.

Enables realistic augmented reality applications like mirror-object insertion.

Effectively reconstructs scene lighting, reflectance, and geometry from panoramic stereo images.

Abstract

We propose a method for estimating high-definition spatially-varying lighting, reflectance, and geometry of a scene from 360$^{\circ}$ stereo images. Our model takes advantage of the 360$^{\circ}$ input to observe the entire scene with geometric detail, then jointly estimates the scene's properties with physical constraints. We first reconstruct a near-field environment light for predicting the lighting at any 3D location within the scene. Then we present a deep learning model that leverages the stereo information to infer the reflectance and surface normal. Lastly, we incorporate the physical constraints between lighting and geometry to refine the reflectance of the scene. Both quantitative and qualitative experiments show that our method, benefiting from the 360$^{\circ}$ observation of the scene, outperforms prior state-of-the-art methods and enables more augmented reality applications such as mirror-objects insertion.