CPTR: Full Transformer Network for Image Captioning

TL;DR

This paper introduces CPTR, a convolution-free Transformer-based model for image captioning that models global context throughout the entire network, outperforming traditional CNN+Transformer approaches on MSCOCO.

Contribution

The paper presents a novel full Transformer architecture for image captioning that eliminates CNNs, enabling global context modeling at all encoder layers.

Findings

Outperforms CNN+Transformer models on MSCOCO

Provides detailed attention visualizations

Demonstrates effectiveness of full Transformer architecture

Abstract

In this paper, we consider the image captioning task from a new sequence-to-sequence prediction perspective and propose CaPtion TransformeR (CPTR) which takes the sequentialized raw images as the input to Transformer. Compared to the "CNN+Transformer" design paradigm, our model can model global context at every encoder layer from the beginning and is totally convolution-free. Extensive experiments demonstrate the effectiveness of the proposed model and we surpass the conventional "CNN+Transformer" methods on the MSCOCO dataset. Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the "words-to-patches" attention in the decoder thanks to the full Transformer architecture.