An Ensemble Model with Attention Based Mechanism for Image Captioning

TL;DR

This paper introduces an ensemble transformer-based model with attention mechanisms for image captioning, achieving state-of-the-art results on Flickr8K and Flickr30K datasets by combining multiple neural networks and selecting captions based on BLEU scores.

Contribution

It proposes a novel ensemble learning framework that enhances image captioning quality by integrating multiple deep neural networks with a voting mechanism based on BLEU scores.

Findings

Achieved highest BLEU scores on Flickr8K and Flickr30K datasets.

Outperformed recent methods in image captioning benchmarks.

Demonstrated the effectiveness of ensemble learning in improving caption quality.

Abstract

Image captioning creates informative text from an input image by creating a relationship between the words and the actual content of an image. Recently, deep learning models that utilize transformers have been the most successful in automatically generating image captions. The capabilities of transformer networks have led to notable progress in several activities related to vision. In this paper, we thoroughly examine transformer models, emphasizing the critical role that attention mechanisms play. The proposed model uses a transformer encoder-decoder architecture to create textual captions and a deep learning convolutional neural network to extract features from the images. To create the captions, we present a novel ensemble learning framework that improves the richness of the generated captions by utilizing several deep neural network architectures based on a voting mechanism that chooses the caption with the highest bilingual evaluation understudy (BLEU) score. The proposed model was evaluated using publicly available datasets. Using the Flickr8K dataset, the proposed model achieved the highest BLEU-[1-3] scores with rates of 0.728, 0.495, and 0.323, respectively. The suggested model outperformed the latest methods in Flickr30k datasets, determined by BLEU-[1-4] scores with rates of 0.798, 0.561, 0.387, and 0.269, respectively. The model efficacy was also obtained by the Semantic propositional image caption evaluation (SPICE) metric with a scoring rate of 0.164 for the Flicker8k dataset and 0.387 for the Flicker30k. Finally, ensemble learning significantly advances the process of image captioning and, hence, can be leveraged in various applications across different domains.