UniGS: Modeling Unitary 3D Gaussians for Novel View Synthesis from Sparse-view Images

TL;DR

UniGS introduces a novel 3D Gaussian modeling approach for high-fidelity novel view synthesis from sparse multi-view images, leveraging a DETR-like framework to improve accuracy and efficiency.

Contribution

The paper proposes modeling unitary 3D Gaussians in world space with a DETR-like multi-view updating mechanism, enabling arbitrary input views without memory issues or retraining.

Findings

Achieves 4.2 dB PSNR improvement on GSO benchmark.

Effectively avoids ghosting artifacts in view synthesis.

Supports arbitrary number of input views without retraining.

Abstract

In this work, we introduce UniGS, a novel 3D Gaussian reconstruction and novel view synthesis model that predicts a high-fidelity representation of 3D Gaussians from arbitrary number of posed sparse-view images. Previous methods often regress 3D Gaussians locally on a per-pixel basis for each view and then transfer them to world space and merge them through point concatenation. In contrast, Our approach involves modeling unitary 3D Gaussians in world space and updating them layer by layer. To leverage information from multi-view inputs for updating the unitary 3D Gaussians, we develop a DETR (DEtection TRansformer)-like framework, which treats 3D Gaussians as queries and updates their parameters by performing multi-view cross-attention (MVDFA) across multiple input images, which are treated as keys and values. This approach effectively avoids `ghosting' issue and allocates more 3D Gaussians to complex regions. Moreover, since the number of 3D Gaussians used as decoder queries is independent of the number of input views, our method allows arbitrary number of multi-view images as input without causing memory explosion or requiring retraining. Extensive experiments validate the advantages of our approach, showcasing superior performance over existing methods quantitatively (improving PSNR by 4.2 dB when trained on Objaverse and tested on the GSO benchmark) and qualitatively. The code will be released at https://github.com/jwubz123/UNIG.