ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes

TL;DR

ScanEnts3D is a large-scale dataset that links natural language to 3D objects, significantly enhancing visio-linguistic models' performance and interpretability in 3D scene understanding tasks.

Contribution

The paper introduces ScanEnts3D, a new dataset with explicit object-phrase correspondences, and demonstrates its effectiveness in improving neural models for 3D language understanding and generation.

Findings

Improves state-of-the-art in Nr3D and ScanRefer by 4.3% and 5.0%.

Enhances 3D neural speaker performance by 13.2 CIDEr points.

Supports better generalization and interpretability of visio-linguistic models.

Abstract

The two popular datasets ScanRefer [16] and ReferIt3D [3] connect natural language to real-world 3D data. In this paper, we curate a large-scale and complementary dataset extending both the aforementioned ones by associating all objects mentioned in a referential sentence to their underlying instances inside a 3D scene. Specifically, our Scan Entities in 3D (ScanEnts3D) dataset provides explicit correspondences between 369k objects across 84k natural referential sentences, covering 705 real-world scenes. Crucially, we show that by incorporating intuitive losses that enable learning from this novel dataset, we can significantly improve the performance of several recently introduced neural listening architectures, including improving the SoTA in both the Nr3D and ScanRefer benchmarks by 4.3% and 5.0%, respectively. Moreover, we experiment with competitive baselines and recent methods for the task of language generation and show that, as with neural listeners, 3D neural speakers can also noticeably benefit by training with ScanEnts3D, including improving the SoTA by 13.2 CIDEr points on the Nr3D benchmark. Overall, our carefully conducted experimental studies strongly support the conclusion that, by learning on ScanEnts3D, commonly used visio-linguistic 3D architectures can become more efficient and interpretable in their generalization without needing to provide these newly collected annotations at test time. The project's webpage is https://scanents3d.github.io/ .