End-to-End 3D Dense Captioning with Vote2Cap-DETR

TL;DR

This paper introduces Vote2Cap-DETR, a transformer-based framework for 3D dense captioning that simplifies the pipeline by integrating detection and captioning into a single stage, achieving state-of-the-art results.

Contribution

The paper presents a full transformer encoder-decoder architecture with learnable vote queries for 3D dense captioning, eliminating the need for hand-crafted components and two-stage detection-captioning pipelines.

Findings

Surpasses state-of-the-art by 11.13% CIDEr@0.5IoU on ScanRefer

Achieves 7.11% improvement on Nr3D dataset

Demonstrates effectiveness of a unified transformer framework for 3D dense captioning

Abstract

3D dense captioning aims to generate multiple captions localized with their associated object regions. Existing methods follow a sophisticated ``detect-then-describe'' pipeline equipped with numerous hand-crafted components. However, these hand-crafted components would yield suboptimal performance given cluttered object spatial and class distributions among different scenes. In this paper, we propose a simple-yet-effective transformer framework Vote2Cap-DETR based on recent popular \textbf{DE}tection \textbf{TR}ansformer (DETR). Compared with prior arts, our framework has several appealing advantages: 1) Without resorting to numerous hand-crafted components, our method is based on a full transformer encoder-decoder architecture with a learnable vote query driven object decoder, and a caption decoder that produces the dense captions in a set-prediction manner. 2) In contrast to the two-stage scheme, our method can perform detection and captioning in one-stage. 3) Without bells and whistles, extensive experiments on two commonly used datasets, ScanRefer and Nr3D, demonstrate that our Vote2Cap-DETR surpasses current state-of-the-arts by 11.13\% and 7.11\% in CIDEr@0.5IoU, respectively. Codes will be released soon.