SpecGaussian with Latent Features: A High-quality Modeling of the View-dependent Appearance for 3D Gaussian Splatting

TL;DR

This paper introduces Latent-SpecGS, a novel 3D Gaussian Splatting method that effectively models view-dependent specular reflections and complex appearances, improving rendering quality in novel view synthesis.

Contribution

It proposes a universal latent neural descriptor and dual CNN decoders to better represent appearance and geometry, overcoming limitations of spherical harmonics in 3D-GS.

Findings

Achieves competitive novel view synthesis results.

Effectively models specular reflections in complex scenes.

Extends 3D-GS capabilities to handle intricate view-dependent appearances.

Abstract

Recently, the 3D Gaussian Splatting (3D-GS) method has achieved great success in novel view synthesis, providing real-time rendering while ensuring high-quality rendering results. However, this method faces challenges in modeling specular reflections and handling anisotropic appearance components, especially in dealing with view-dependent color under complex lighting conditions. Additionally, 3D-GS uses spherical harmonic to learn the color representation, which has limited ability to represent complex scenes. To overcome these challenges, we introduce Lantent-SpecGS, an approach that utilizes a universal latent neural descriptor within each 3D Gaussian. This enables a more effective representation of 3D feature fields, including appearance and geometry. Moreover, two parallel CNNs are designed to decoder the splatting feature maps into diffuse color and specular color separately. A mask that depends on the viewpoint is learned to merge these two colors, resulting in the final rendered image. Experimental results demonstrate that our method obtains competitive performance in novel view synthesis and extends the ability of 3D-GS to handle intricate scenarios with specular reflections.