Image Captioning In the Transformer Age

TL;DR

This paper discusses the transition of image captioning from CNN-RNN architectures to Transformer-based models, emphasizing end-to-end training and the role of self-supervised learning in enhancing IC performance.

Contribution

It provides a survey of recent advances in Transformer-based image captioning and analyzes the significance of IC amidst large-scale self-supervised models.

Findings

Transformers enable homogeneous, end-to-end trainable IC models.

Self-supervised pre-training enhances IC performance.

IC maintains unique importance despite large-scale model developments.

Abstract

Image Captioning (IC) has achieved astonishing developments by incorporating various techniques into the CNN-RNN encoder-decoder architecture. However, since CNN and RNN do not share the basic network component, such a heterogeneous pipeline is hard to be trained end-to-end where the visual encoder will not learn anything from the caption supervision. This drawback inspires the researchers to develop a homogeneous architecture that facilitates end-to-end training, for which Transformer is the perfect one that has proven its huge potential in both vision and language domains and thus can be used as the basic component of the visual encoder and language decoder in an IC pipeline. Meantime, self-supervised learning releases the power of the Transformer architecture that a pre-trained large-scale one can be generalized to various tasks including IC. The success of these large-scale models seems to weaken the importance of the single IC task. However, we demonstrate that IC still has its specific significance in this age by analyzing the connections between IC with some popular self-supervised learning paradigms. Due to the page limitation, we only refer to highly important papers in this short survey and more related works can be found at https://github.com/SjokerLily/awesome-image-captioning.