NRGS: Neural Regularization for Robust 3D Semantic Gaussian Splatting

TL;DR

This paper introduces a neural regularization technique that refines noisy 3D semantic fields from multi-view inconsistent 2D features, improving the accuracy and robustness of 3D Gaussian Splatting.

Contribution

We develop a variance-aware conditional MLP that directly corrects semantic errors in 3D Gaussians, offering an efficient alternative to previous multi-view consistency methods.

Findings

Enhances 3D semantic accuracy across datasets.

Reduces noise in 3D Gaussian representations.

Operates efficiently without extensive preprocessing.

Abstract

We propose a neural regularization method that refines the noisy 3D semantic field produced by lifting multi-view inconsistent 2D features, in order to obtain an accurate and robust 3D semantic Gaussian Splatting. The 2D features extracted from vision foundation models suffer from multi-view inconsistency due to a lack of cross-view constraints. Lifting these inconsistent features directly into 3D Gaussians results in a noisy semantic field, which degrades the performance of downstream tasks. Previous methods either focus on obtaining consistent multi-view features in the preprocessing stage or aim to mitigate noise through improved optimization strategies, often at the cost of increased preprocessing time or expensive computational overhead. In contrast, we introduce a variance-aware conditional MLP that operates directly on the 3D Gaussians, leveraging their geometric and appearance attributes to correct semantic errors in 3D space. Experiments on different datasets show that our method enhances the accuracy of lifted semantics, providing an efficient and effective approach to robust 3D semantic Gaussian Splatting.