GSsplat: Generalizable Semantic Gaussian Splatting for Novel-view Synthesis in 3D Scenes

TL;DR

GSsplat introduces a fast, generalizable method for semantic 3D scene synthesis from multiple viewpoints, outperforming existing techniques in speed and segmentation quality.

Contribution

The paper presents GSsplat, a novel semantic Gaussian Splatting approach that improves efficiency and performance in novel-view synthesis for 3D scenes.

Findings

Achieves state-of-the-art semantic synthesis performance.

Operates at the fastest speed among comparable methods.

Effectively predicts scene-adaptive Gaussian parameters.

Abstract

The semantic synthesis of unseen scenes from multiple viewpoints is crucial for research in 3D scene understanding. Current methods are capable of rendering novel-view images and semantic maps by reconstructing generalizable Neural Radiance Fields. However, they often suffer from limitations in speed and segmentation performance. We propose a generalizable semantic Gaussian Splatting method (GSsplat) for efficient novel-view synthesis. Our model predicts the positions and attributes of scene-adaptive Gaussian distributions from once input, replacing the densification and pruning processes of traditional scene-specific Gaussian Splatting. In the multi-task framework, a hybrid network is designed to extract color and semantic information and predict Gaussian parameters. To augment the spatial perception of Gaussians for high-quality rendering, we put forward a novel offset learning module through group-based supervision and a point-level interaction module with spatial unit aggregation. When evaluated with varying numbers of multi-view inputs, GSsplat achieves state-of-the-art performance for semantic synthesis at the fastest speed.