Space3D-Bench: Spatial 3D Question Answering Benchmark

TL;DR

Space3D-Bench introduces a comprehensive 3D question-answering dataset with diverse modalities and an assessment system, advancing the evaluation of models' understanding of spatial environments.

Contribution

It provides a large, balanced dataset of 3D spatial questions across multiple modalities and a novel assessment system using vision-language models.

Findings

Baseline RAG3D-Chat achieves 67% accuracy.

Dataset covers a wide range of 3D spatial reasoning tasks.

Assessment system effectively grades natural language responses.

Abstract

Answering questions about the spatial properties of the environment poses challenges for existing language and vision foundation models due to a lack of understanding of the 3D world notably in terms of relationships between objects. To push the field forward, multiple 3D Q&A datasets were proposed which, overall, provide a variety of questions, but they individually focus on particular aspects of 3D reasoning or are limited in terms of data modalities. To address this, we present Space3D-Bench - a collection of 1000 general spatial questions and answers related to scenes of the Replica dataset which offers a variety of data modalities: point clouds, posed RGB-D images, navigation meshes and 3D object detections. To ensure that the questions cover a wide range of 3D objectives, we propose an indoor spatial questions taxonomy inspired by geographic information systems and use it to balance the dataset accordingly. Moreover, we provide an assessment system that grades natural language responses based on predefined ground-truth answers by leveraging a Vision Language Model's comprehension of both text and images to compare the responses with ground-truth textual information or relevant visual data. Finally, we introduce a baseline called RAG3D-Chat integrating the world understanding of foundation models with rich context retrieval, achieving an accuracy of 67% on the proposed dataset.