Image Captioning using Multiple Transformers for Self-Attention Mechanism

TL;DR

This paper introduces MTSM, a novel image captioning method using multiple transformers to improve real-time captioning accuracy by capturing local and global object relationships.

Contribution

It proposes a new transformer-based framework that integrates region proposals and self-attention for enhanced image captioning performance.

Findings

Achieves improved captioning accuracy on MSCOCO dataset.

Effectively models local and global object relationships.

Demonstrates competitive real-time captioning capabilities.

Abstract

Real-time image captioning, along with adequate precision, is the main challenge of this research field. The present work, Multiple Transformers for Self-Attention Mechanism (MTSM), utilizes multiple transformers to address these problems. The proposed algorithm, MTSM, acquires region proposals using a transformer detector (DETR). Consequently, MTSM achieves the self-attention mechanism by transferring these region proposals and their visual and geometrical features through another transformer and learns the objects' local and global interconnections. The qualitative and quantitative results of the proposed algorithm, MTSM, are shown on the MSCOCO dataset.