Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding

TL;DR

This paper introduces a joint semantic and rendering enhancement framework for 3D Gaussian modeling that uses anisotropic descriptors and local signals to improve detail capture, efficiency, and robustness across scenes.

Contribution

It proposes a novel anisotropic 3D Gaussian descriptor and a joint enhancement framework that integrates semantic and rendering branches with adaptive resource allocation and cross-scene knowledge transfer.

Findings

Improved segmentation accuracy and rendering quality.

Enhanced efficiency through selective resource allocation.

Faster convergence and robust representations across scenes.

Abstract

Recent works propose extending 3DGS with semantic feature vectors for simultaneous semantic segmentation and image rendering. However, these methods often treat the semantic and rendering branches separately, relying solely on 2D supervision while ignoring the 3D Gaussian geometry. Moreover, current adaptive strategies adapt the Gaussian set depending solely on rendering gradients, which can be insufficient in subtle or textureless regions. In this work, we propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches. Firstly, unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor using the Laplace-Beltrami operator to capture fine-grained 3D shape details, thereby distinguishing objects with similar appearances and reducing reliance on potentially noisy 2D guidance. In addition, without relying solely on rendering gradient, we adaptively adjust Gaussian allocation and spherical harmonics with local semantic and shape signals, enhancing rendering efficiency through selective resource allocation. Finally, we employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations without relearning shape information from scratch for each new scene. Experiments on multiple datasets demonstrate improvements in segmentation accuracy and rendering quality while maintaining high rendering frame rates.