Camera-Aware Cross-View Alignment for Referring 3D Gaussian Splatting Segmentation

TL;DR

This paper introduces CaRF, a novel camera-aware framework for 3D Gaussian splatting segmentation that enhances view consistency and grounding accuracy in language-based 3D scene understanding.

Contribution

CaRF incorporates camera geometry into Gaussian interactions and aligns cross-view responses, enabling geometry-aware, view-consistent reasoning in 3D segmentation.

Findings

Achieves state-of-the-art performance on three benchmarks.

Improves mIoU by up to 16.8% over previous methods.

Demonstrates effective cross-view alignment and reasoning.

Abstract

Referring 3D Gaussian Splatting Segmentation (R3DGS) aims to ground free-form language queries in 3D Gaussian fields. However, existing methods rely on single-view pseudo supervision, leading to viewpoint drift and inconsistent predictions across views. We propose CaRF (Camera-aware Referring Field), a camera-aware cross-view alignment framework for view-consistent referring in 3D Gaussian splatting. CaRF introduces Camera-conditioned Alignment Modulation (CAM) to inject camera geometry into Gaussian-text interactions, and Gaussian-level Cross-view Logit Alignment (GCLA) to explicitly align referring responses of the same Gaussians across calibrated views during training. By turning cross-view discrepancy into an optimizable objective, CaRF enables geometry-aware and view-consistent reasoning directly in the Gaussian space. Extensive experiments on three benchmarks demonstrate that CaRF achieves state-of-the-art performance, improving mIoU by 16.8%, 4.3%, and 2.0% on Ref-LERF, LERF-OVS, and 3D-OVS, respectively. Our code is available at https://github.com/eR3R3/CaRF.