Phys3DGS: Physically-based 3D Gaussian Splatting for Inverse Rendering

TL;DR

This paper introduces Phys3DGS, a novel 3D Gaussian splatting method for inverse rendering that combines deferred rendering and mesh-based representations to improve quality and enable real-time relighting.

Contribution

It proposes a two-step training approach with mesh extraction and regularization, addressing hidden Gaussian issues and enhancing inverse rendering quality.

Findings

Significantly better rendering quality under relighting.

Outperforms existing 3DGS-based inverse rendering methods.

Achieves real-time rendering comparable to voxel grid methods.

Abstract

We propose two novel ideas (adoption of deferred rendering and mesh-based representation) to improve the quality of 3D Gaussian splatting (3DGS) based inverse rendering. We first report a problem incurred by hidden Gaussians, where Gaussians beneath the surface adversely affect the pixel color in the volume rendering adopted by the existing methods. In order to resolve the problem, we propose applying deferred rendering and report new problems incurred in a naive application of deferred rendering to the existing 3DGS-based inverse rendering. In an effort to improve the quality of 3DGS-based inverse rendering under deferred rendering, we propose a novel two-step training approach which (1) exploits mesh extraction and utilizes a hybrid mesh-3DGS representation and (2) applies novel regularization methods to better exploit the mesh. Our experiments show that, under relighting, the proposed method offers significantly better rendering quality than the existing 3DGS-based inverse rendering methods. Compared with the SOTA voxel grid-based inverse rendering method, it gives better rendering quality while offering real-time rendering.