Gaussian Splatting with NeRF-based Color and Opacity

TL;DR

This paper introduces a hybrid 3D rendering model combining Gaussian Splatting and NeRF techniques, enabling efficient, high-quality rendering with improved shadow, reflection, and transparency effects without complex textures.

Contribution

The paper proposes Viewing Direction Gaussian Splatting (VDGS), a novel hybrid model that integrates Gaussian Splatting with NeRF-based color and opacity encoding for enhanced 3D rendering.

Findings

Faster training and inference compared to traditional NeRFs

Improved rendering of shadows, reflections, and transparency

Reduces the number of Gaussian components needed for conditioning

Abstract

Neural Radiance Fields (NeRFs) have demonstrated the remarkable potential of neural networks to capture the intricacies of 3D objects. By encoding the shape and color information within neural network weights, NeRFs excel at producing strikingly sharp novel views of 3D objects. Recently, numerous generalizations of NeRFs utilizing generative models have emerged, expanding its versatility. In contrast, Gaussian Splatting (GS) offers a similar render quality with faster training and inference as it does not need neural networks to work. It encodes information about the 3D objects in the set of Gaussian distributions that can be rendered in 3D similarly to classical meshes. Unfortunately, GS are difficult to condition since they usually require circa hundred thousand Gaussian components. To mitigate the caveats of both models, we propose a hybrid model Viewing Direction Gaussian Splatting (VDGS) that uses GS representation of the 3D object's shape and NeRF-based encoding of color and opacity. Our model uses Gaussian distributions with trainable positions (i.e. means of Gaussian), shape (i.e. covariance of Gaussian), color and opacity, and a neural network that takes Gaussian parameters and viewing direction to produce changes in the said color and opacity. As a result, our model better describes shadows, light reflections, and the transparency of 3D objects without adding additional texture and light components.