SemGS: Feed-Forward Semantic 3D Gaussian Splatting from Sparse Views for Generalizable Scene Understanding

TL;DR

SemGS is a fast, generalizable framework that reconstructs semantic 3D scenes from sparse views using a dual-branch architecture, camera-aware attention, and Gaussian-based decoding, outperforming existing methods.

Contribution

We introduce SemGS, a novel feed-forward approach for semantic 3D scene understanding from sparse views, with a dual-branch architecture and camera-aware attention for improved generalization.

Findings

Achieves state-of-the-art results on benchmark datasets.

Provides rapid inference suitable for real-world applications.

Demonstrates strong generalization across synthetic and real-world scenes.

Abstract

Semantic understanding of 3D scenes is essential for robots to operate effectively and safely in complex environments. Existing methods for semantic scene reconstruction and semantic-aware novel view synthesis often rely on dense multi-view inputs and require scene-specific optimization, limiting their practicality and scalability in real-world applications. To address these challenges, we propose SemGS, a feed-forward framework for reconstructing generalizable semantic fields from sparse image inputs. SemGS uses a dual-branch architecture to extract color and semantic features, where the two branches share shallow CNN layers, allowing semantic reasoning to leverage textural and structural cues in color appearance. We also incorporate a camera-aware attention mechanism into the feature extractor to explicitly model geometric relationships between camera viewpoints. The extracted features are decoded into dual-Gaussians that share geometric consistency while preserving branch-specific attributes, and further rasterized to synthesize semantic maps under novel viewpoints. Additionally, we introduce a regional smoothness loss to enhance semantic coherence. Experiments show that SemGS achieves state-of-the-art performance on benchmark datasets, while providing rapid inference and strong generalization capabilities across diverse synthetic and real-world scenarios.