Gaussian Time Machine: A Real-Time Rendering Methodology for Time-Variant Appearances

TL;DR

Gaussian Time Machine introduces a real-time rendering method that models time-variant appearances using Gaussian primitives and a lightweight MLP, achieving high fidelity and fast rendering in dynamic scenes with changing weather and lighting conditions.

Contribution

The paper proposes a novel approach combining Gaussian primitives with time embedding and a decomposed color model for efficient, accurate, and smooth rendering of dynamic scenes.

Findings

Achieved state-of-the-art rendering fidelity on three datasets.

100x faster rendering compared to NeRF-based methods.

Successfully disentangles appearance changes and enables smooth interpolation.

Abstract

Recent advancements in neural rendering techniques have significantly enhanced the fidelity of 3D reconstruction. Notably, the emergence of 3D Gaussian Splatting (3DGS) has marked a significant milestone by adopting a discrete scene representation, facilitating efficient training and real-time rendering. Several studies have successfully extended the real-time rendering capability of 3DGS to dynamic scenes. However, a challenge arises when training images are captured under vastly differing weather and lighting conditions. This scenario poses a challenge for 3DGS and its variants in achieving accurate reconstructions. Although NeRF-based methods (NeRF-W, CLNeRF) have shown promise in handling such challenging conditions, their computational demands hinder real-time rendering capabilities. In this paper, we present Gaussian Time Machine (GTM) which models the time-dependent attributes of Gaussian primitives with discrete time embedding vectors decoded by a lightweight Multi-Layer-Perceptron(MLP). By adjusting the opacity of Gaussian primitives, we can reconstruct visibility changes of objects. We further propose a decomposed color model for improved geometric consistency. GTM achieved state-of-the-art rendering fidelity on 3 datasets and is 100 times faster than NeRF-based counterparts in rendering. Moreover, GTM successfully disentangles the appearance changes and renders smooth appearance interpolation.