TSGaussian: Semantic and Depth-Guided Target-Specific Gaussian Splatting from Sparse Views

TL;DR

TSGaussian is a novel framework that enhances 3D scene reconstruction from sparse views by integrating semantic and depth information, focusing on target-specific accuracy and efficiency in novel view synthesis.

Contribution

The paper introduces TSGaussian, combining semantic constraints, depth priors, and a clustering/pruning strategy to improve target-specific 3D reconstruction from sparse views.

Findings

Outperforms state-of-the-art methods on multiple datasets.

Effectively reconstructs complex targets with reduced redundancy.

Achieves superior novel view synthesis accuracy.

Abstract

Recent advances in Gaussian Splatting have significantly advanced the field, achieving both panoptic and interactive segmentation of 3D scenes. However, existing methodologies often overlook the critical need for reconstructing specified targets with complex structures from sparse views. To address this issue, we introduce TSGaussian, a novel framework that combines semantic constraints with depth priors to avoid geometry degradation in challenging novel view synthesis tasks. Our approach prioritizes computational resources on designated targets while minimizing background allocation. Bounding boxes from YOLOv9 serve as prompts for Segment Anything Model to generate 2D mask predictions, ensuring semantic accuracy and cost efficiency. TSGaussian effectively clusters 3D gaussians by introducing a compact identity encoding for each Gaussian ellipsoid and incorporating 3D spatial consistency regularization. Leveraging these modules, we propose a pruning strategy to effectively reduce redundancy in 3D gaussians. Extensive experiments demonstrate that TSGaussian outperforms state-of-the-art methods on three standard datasets and a new challenging dataset we collected, achieving superior results in novel view synthesis of specific objects. Code is available at: https://github.com/leon2000-ai/TSGaussian.