Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds

TL;DR

This paper introduces SpaCap3D, a transformer-based model that leverages spatial relations to improve 3D dense captioning of objects in point cloud scenes, outperforming existing methods.

Contribution

The paper proposes a novel spatiality-guided encoder-decoder architecture for 3D dense captioning, emphasizing spatial relation learning to enhance caption accuracy.

Findings

Outperforms baseline by 4.94% and 9.61% in CIDEr@0.5IoU on two datasets

Introduces spatial relation learning in 3D captioning models

Demonstrates the effectiveness of spatiality-guided encoding in scene understanding

Abstract

Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest. To detect and describe objects in a scene, following the spirit of neural machine translation, we propose a transformer-based encoder-decoder architecture, namely SpaCap3D, to transform objects into descriptions, where we especially investigate the relative spatiality of objects in 3D scenes and design a spatiality-guided encoder via a token-to-token spatial relation learning objective and an object-centric decoder for precise and spatiality-enhanced object caption generation. Evaluated on two benchmark datasets, ScanRefer and ReferIt3D, our proposed SpaCap3D outperforms the baseline method Scan2Cap by 4.94% and 9.61% in CIDEr@0.5IoU, respectively. Our project page with source code and supplementary files is available at https://SpaCap3D.github.io/ .