GaussianPainter: Painting Point Cloud into 3D Gaussians with Normal Guidance

TL;DR

GaussianPainter is a novel feed-forward method that converts point clouds into 3D Gaussians using normal guidance and appearance injection, enabling efficient and high-quality 3D content creation from reference images.

Contribution

It introduces a surface normal-based rotation estimation and an appearance injection module to improve 3D Gaussian painting from point clouds in a single pass.

Findings

Achieves high-quality 3D Gaussian generation efficiently.

Balances fidelity and diversity in 3D content creation.

Outperforms optimization-based methods in speed and robustness.

Abstract

In this paper, we present GaussianPainter, the first method to paint a point cloud into 3D Gaussians given a reference image. GaussianPainter introduces an innovative feed-forward approach to overcome the limitations of time-consuming test-time optimization in 3D Gaussian splatting. Our method addresses a critical challenge in the field: the non-uniqueness problem inherent in the large parameter space of 3D Gaussian splatting. This space, encompassing rotation, anisotropic scales, and spherical harmonic coefficients, introduces the challenge of rendering similar images from substantially different Gaussian fields. As a result, feed-forward networks face instability when attempting to directly predict high-quality Gaussian fields, struggling to converge on consistent parameters for a given output. To address this issue, we propose to estimate a surface normal for each point to determine its Gaussian rotation. This strategy enables the network to effectively predict the remaining Gaussian parameters in the constrained space. We further enhance our approach with an appearance injection module, incorporating reference image appearance into Gaussian fields via a multiscale triplane representation. Our method successfully balances efficiency and fidelity in 3D Gaussian generation, achieving high-quality, diverse, and robust 3D content creation from point clouds in a single forward pass.