RTR-GS: 3D Gaussian Splatting for Inverse Rendering with Radiance Transfer and Reflection

TL;DR

RTR-GS is a novel inverse rendering framework that improves rendering of reflective objects by decomposing BRDF and lighting, enabling realistic relighting and view synthesis from multi-view images.

Contribution

It introduces a hybrid rendering model combining forward and deferred rendering to handle reflections and high-frequency details effectively.

Findings

Enhanced novel view synthesis and relighting capabilities.

Improved BRDF and lighting decomposition accuracy.

Maintains efficient training and inference processes.

Abstract

3D Gaussian Splatting (3DGS) has demonstrated impressive capabilities in novel view synthesis. However, rendering reflective objects remains a significant challenge, particularly in inverse rendering and relighting. We introduce RTR-GS, a novel inverse rendering framework capable of robustly rendering objects with arbitrary reflectance properties, decomposing BRDF and lighting, and delivering credible relighting results. Given a collection of multi-view images, our method effectively recovers geometric structure through a hybrid rendering model that combines forward rendering for radiance transfer with deferred rendering for reflections. This approach successfully separates high-frequency and low-frequency appearances, mitigating floating artifacts caused by spherical harmonic overfitting when handling high-frequency details. We further refine BRDF and lighting decomposition using an additional physically-based deferred rendering branch. Experimental results show that our method enhances novel view synthesis, normal estimation, decomposition, and relighting while maintaining efficient training inference process.