RefGaussian: Disentangling Reflections from 3D Gaussian Splatting for Realistic Rendering

TL;DR

RefGaussian enhances 3D Gaussian Splatting by disentangling reflections into transmitted and reflected components using spherical harmonics, leading to more realistic rendering, improved scene reconstruction, and enabling scene editing with physical coherence.

Contribution

We introduce a novel method to separate reflections from scenes in 3D-GS using spherical harmonics and local regularization, improving realism and reconstruction accuracy.

Findings

Achieves superior novel view synthesis results.

Provides more accurate depth estimation.

Enables scene editing with high quality and physical consistency.

Abstract

3D Gaussian Splatting (3D-GS) has made a notable advancement in the field of neural rendering, 3D scene reconstruction, and novel view synthesis. Nevertheless, 3D-GS encounters the main challenge when it comes to accurately representing physical reflections, especially in the case of total reflection and semi-reflection that are commonly found in real-world scenes. This limitation causes reflections to be mistakenly treated as independent elements with physical presence, leading to imprecise reconstructions. Herein, to tackle this challenge, we propose RefGaussian to disentangle reflections from 3D-GS for realistically modeling reflections. Specifically, we propose to split a scene into transmitted and reflected components and represent these components using two Spherical Harmonics (SH). Given that this decomposition is not fully determined, we employ local regularization techniques to ensure local smoothness for both the transmitted and reflected components, thereby achieving more plausible decomposition outcomes than 3D-GS. Experimental results demonstrate that our approach achieves superior novel view synthesis and accurate depth estimation outcomes. Furthermore, it enables the utilization of scene editing applications, ensuring both high-quality results and physical coherence.