Dual-Level Collaborative Transformer for Image Captioning

TL;DR

This paper introduces a Dual-Level Collaborative Transformer that combines region and grid features for image captioning, using novel attention mechanisms and geometric alignment to improve contextual understanding and achieve state-of-the-art results.

Contribution

The paper proposes a novel DLCT network with dual-way self-attention and geometric alignment for better feature fusion in image captioning.

Findings

Achieves new state-of-the-art CIDEr-D score of 133.8% on Karpathy split.

Demonstrates effective fusion of region and grid features.

Outperforms existing methods on MS-COCO dataset.

Abstract

Descriptive region features extracted by object detection networks have played an important role in the recent advancements of image captioning. However, they are still criticized for the lack of contextual information and fine-grained details, which in contrast are the merits of traditional grid features. In this paper, we introduce a novel Dual-Level Collaborative Transformer (DLCT) network to realize the complementary advantages of the two features. Concretely, in DLCT, these two features are first processed by a novelDual-way Self Attenion (DWSA) to mine their intrinsic properties, where a Comprehensive Relation Attention component is also introduced to embed the geometric information. In addition, we propose a Locality-Constrained Cross Attention module to address the semantic noises caused by the direct fusion of these two features, where a geometric alignment graph is constructed to accurately align and reinforce region and grid features. To validate our model, we conduct extensive experiments on the highly competitive MS-COCO dataset, and achieve new state-of-the-art performance on both local and online test sets, i.e., 133.8% CIDEr-D on Karpathy split and 135.4% CIDEr on the official split. Code is available at https://github.com/luo3300612/image-captioning-DLCT.