PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting

TL;DR

PhySG is an inverse rendering framework that reconstructs geometry, materials, and lighting from images using spherical Gaussians and neural signed distance functions, enabling realistic relighting and material editing.

Contribution

It introduces a fully differentiable inverse rendering pipeline utilizing spherical Gaussians and neural SDFs for scene reconstruction and editing from RGB images.

Findings

Effective reconstruction of geometry, materials, and illumination.

Supports physics-based material and lighting editing.

Works on challenging non-Lambertian scenes with natural lighting.

Abstract

We present PhySG, an end-to-end inverse rendering pipeline that includes a fully differentiable renderer and can reconstruct geometry, materials, and illumination from scratch from a set of RGB input images. Our framework represents specular BRDFs and environmental illumination using mixtures of spherical Gaussians, and represents geometry as a signed distance function parameterized as a Multi-Layer Perceptron. The use of spherical Gaussians allows us to efficiently solve for approximate light transport, and our method works on scenes with challenging non-Lambertian reflectance captured under natural, static illumination. We demonstrate, with both synthetic and real data, that our reconstructions not only enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination.