3D Segmentation Using Viewpoint-Dependent Spatial Relationships

TL;DR

This paper introduces a large viewpoint-aware 3D referring segmentation dataset and demonstrates that incorporating explicit viewpoint information significantly improves model performance on viewpoint-dependent spatial relations.

Contribution

The authors create a novel 3D segmentation dataset with viewpoint annotations and show how explicit viewpoint encoding enhances model accuracy in spatial understanding.

Findings

Current models struggle with viewpoint-dependent spatial instructions.

Explicit viewpoint conditioning improves segmentation mIoU from 0.30 to 0.47.

The dataset enables evaluation of viewpoint-aware 3D scene understanding.

Abstract

Recent advances in 3D datasets and multimodal models have greatly improved natural language 3D scene understanding. However, most 3D referring segmentation methods do not explicitly represent the observer viewpoint, making spatial relations such as "left," "right," "front," and "behind" ambiguous and difficult to evaluate. We introduce a viewpoint-aware 3D referring segmentation dataset containing 220k benchmark samples, and scalable to tens of millions of viewpoint-conditioned samples through dense viewpoint sampling. In this dataset, target objects can only be identified through observer-centric spatial relations, making viewpoint-conditioned grounding necessary. We construct the benchmark by leveraging camera poses to automatically annotate observer-centric relations (left/right, front/behind) together with viewpoint-independent relations (above/under). Using this benchmark, we evaluate several existing 3D large multimodal models in a zero-shot setting and find that current models struggle with viewpoint-dependent spatial instructions. We further study how explicit viewpoint information can be incorporated into 3D large multimodal models. We introduce a viewpoint representation that encodes camera poses and conditions the model on the observation viewpoint, improving segmentation accuracy on viewpoint-dependent relations and increasing mIoU from 0.30 to 0.47 compared to a model without viewpoint conditioning. The dataset, code, and trained models will be made publicly available upon acceptance.