LeanGaussian: Breaking Pixel or Point Cloud Correspondence in Modeling 3D Gaussians

TL;DR

LeanGaussian introduces a novel 3D Gaussian modeling approach that abandons pixel or point cloud correspondence constraints, leading to more accurate geometry, textures, and efficient rendering in novel view synthesis.

Contribution

It proposes breaking the pixel or point cloud correspondence in Gaussian modeling using deformable Transformers, improving 3D reconstruction and rendering performance.

Findings

Outperforms prior methods by ~6.1% in PSNR on ShapeNet and Google Scanned Objects datasets.

Achieves 7.2 FPS 3D reconstruction and 500 FPS rendering speeds.

Demonstrates improved geometry and texture quality in novel view synthesis.

Abstract

Recently, Gaussian splatting has demonstrated significant success in novel view synthesis. Current methods often regress Gaussians with pixel or point cloud correspondence, linking each Gaussian with a pixel or a 3D point. This leads to the redundancy of Gaussians being used to overfit the correspondence rather than the objects represented by the 3D Gaussians themselves, consequently wasting resources and lacking accurate geometries or textures. In this paper, we introduce LeanGaussian, a novel approach that treats each query in deformable Transformer as one 3D Gaussian ellipsoid, breaking the pixel or point cloud correspondence constraints. We leverage deformable decoder to iteratively refine the Gaussians layer-by-layer with the image features as keys and values. Notably, the center of each 3D Gaussian is defined as 3D reference points, which are then projected onto the image for deformable attention in 2D space. On both the ShapeNet SRN dataset (category level) and the Google Scanned Objects dataset (open-category level, trained with the Objaverse dataset), our approach, outperforms prior methods by approximately 6.1%, achieving a PSNR of 25.44 and 22.36, respectively. Additionally, our method achieves a 3D reconstruction speed of 7.2 FPS and rendering speed 500 FPS. Codes are available at https://github.com/jwubz123/LeanGaussian.