OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding

TL;DR

OpenScan introduces a comprehensive benchmark for evaluating generalized open-vocabulary 3D scene understanding, emphasizing the need for models to interpret diverse linguistic attributes beyond object classes.

Contribution

The paper proposes the GOV-3D task and introduces the OpenScan benchmark, expanding evaluation to include fine-grained attributes like affordance, property, and material.

Findings

State-of-the-art OV-3D methods perform poorly on GOV-3D tasks.

Existing methods struggle with abstract vocabulary understanding.

Scaling object classes alone is insufficient for scene comprehension.

Abstract

Open-vocabulary 3D scene understanding (OV-3D) aims to localize and classify novel objects beyond the closed set of object classes. However, existing approaches and benchmarks primarily focus on the open vocabulary problem within the context of object classes, which is insufficient in providing a holistic evaluation to what extent a model understands the 3D scene. In this paper, we introduce a more challenging task called Generalized Open-Vocabulary 3D Scene Understanding (GOV-3D) to explore the open vocabulary problem beyond object classes. It encompasses an open and diverse set of generalized knowledge, expressed as linguistic queries of fine-grained and object-specific attributes. To this end, we contribute a new benchmark named \textit{OpenScan}, which consists of 3D object attributes across eight representative linguistic aspects, including affordance, property, and material. We further evaluate state-of-the-art OV-3D methods on our OpenScan benchmark and discover that these methods struggle to comprehend the abstract vocabularies of the GOV-3D task, a challenge that cannot be addressed simply by scaling up object classes during training. We highlight the limitations of existing methodologies and explore promising directions to overcome the identified shortcomings.