HybridGS: Decoupling Transients and Statics with 2D and 3D Gaussian Splatting

TL;DR

HybridGS introduces a novel scene representation that decouples transient objects from static scenes using 2D and 3D Gaussian splatting, improving view synthesis quality in complex scenes.

Contribution

The paper proposes HybridGS, a hybrid scene representation combining 2D and 3D Gaussians, along with a multi-view supervision method and multi-stage training for better rendering of scenes with transients.

Findings

State-of-the-art view synthesis performance on benchmark datasets.

Effective separation of transient and static scene components.

Robustness in indoor and outdoor scenes with distracting elements.

Abstract

Generating high-quality novel view renderings of 3D Gaussian Splatting (3DGS) in scenes featuring transient objects is challenging. We propose a novel hybrid representation, termed as HybridGS, using 2D Gaussians for transient objects per image and maintaining traditional 3D Gaussians for the whole static scenes. Note that, the 3DGS itself is better suited for modeling static scenes that assume multi-view consistency, but the transient objects appear occasionally and do not adhere to the assumption, thus we model them as planar objects from a single view, represented with 2D Gaussians. Our novel representation decomposes the scene from the perspective of fundamental viewpoint consistency, making it more reasonable. Additionally, we present a novel multi-view regulated supervision method for 3DGS that leverages information from co-visible regions, further enhancing the distinctions between the transients and statics. Then, we propose a straightforward yet effective multi-stage training strategy to ensure robust training and high-quality view synthesis across various settings. Experiments on benchmark datasets show our state-of-the-art performance of novel view synthesis in both indoor and outdoor scenes, even in the presence of distracting elements.