3DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset

TL;DR

This paper introduces 3DBench, a comprehensive 3D benchmark and instruction-tuning dataset designed to evaluate multi-modal large language models across spatial, semantic, perception, and planning tasks, addressing current evaluation gaps.

Contribution

The paper presents a scalable 3D benchmark and a large-scale instruction-tuning dataset covering diverse multi-modal tasks, enabling thorough assessment of MLLMs' spatial understanding and expressive capabilities.

Findings

3DBench outperforms existing datasets in evaluating MLLMs.

The dataset covers over 0.23 million QA pairs across 10 tasks.

Experiments reveal current MLLMs have notable limitations in spatial reasoning.

Abstract

Evaluating the performance of Multi-modal Large Language Models (MLLMs), integrating both point cloud and language, presents significant challenges. The lack of a comprehensive assessment hampers determining whether these models truly represent advancements, thereby impeding further progress in the field. Current evaluations heavily rely on classification and caption tasks, falling short in providing a thorough assessment of MLLMs. A pressing need exists for a more sophisticated evaluation method capable of thoroughly analyzing the spatial understanding and expressive capabilities of these models. To address these issues, we introduce a scalable 3D benchmark, accompanied by a large-scale instruction-tuning dataset known as 3DBench, providing an extensible platform for a comprehensive evaluation of MLLMs. Specifically, we establish the benchmark that spans a wide range of spatial and semantic scales, from object-level to scene-level, addressing both perception and planning tasks. Furthermore, we present a rigorous pipeline for automatically constructing scalable 3D instruction-tuning datasets, covering 10 diverse multi-modal tasks with more than 0.23 million QA pairs generated in total. Thorough experiments evaluating trending MLLMs, comparisons against existing datasets, and variations of training protocols demonstrate the superiority of 3DBench, offering valuable insights into current limitations and potential research directions.