PointGS: Point Attention-Aware Sparse View Synthesis with Gaussian Splatting

TL;DR

PointGS introduces a novel sparse-view 3D scene rendering method that combines Gaussian splatting with point-wise attention, enabling real-time, high-quality visualization even with limited input views.

Contribution

The paper presents a point-wise feature-aware Gaussian splatting framework that improves sparse view synthesis by integrating a self-attention mechanism for enhanced point features.

Findings

Outperforms NeRF-based methods in rendering speed and quality.

Achieves competitive results with state-of-the-art 3DGS approaches in few-shot scenarios.

Enables real-time rendering from sparse inputs with high visual fidelity.

Abstract

3D Gaussian splatting (3DGS) is an innovative rendering technique that surpasses the neural radiance field (NeRF) in both rendering speed and visual quality by leveraging an explicit 3D scene representation. Existing 3DGS approaches require a large number of calibrated views to generate a consistent and complete scene representation. When input views are limited, 3DGS tends to overfit the training views, leading to noticeable degradation in rendering quality. To address this limitation, we propose a Point-wise Feature-Aware Gaussian Splatting framework that enables real-time, high-quality rendering from sparse training views. Specifically, we first employ the latest stereo foundation model to estimate accurate camera poses and reconstruct a dense point cloud for Gaussian initialization. We then encode the colour attributes of each 3D Gaussian by sampling and aggregating multiscale 2D appearance features from sparse inputs. To enhance point-wise appearance representation, we design a point interaction network based on a self-attention mechanism, allowing each Gaussian point to interact with its nearest neighbors. These enriched features are subsequently decoded into Gaussian parameters through two lightweight multi-layer perceptrons (MLPs) for final rendering. Extensive experiments on diverse benchmarks demonstrate that our method significantly outperforms NeRF-based approaches and achieves competitive performance under few-shot settings compared to the state-of-the-art 3DGS methods.