Improved Bengali Image Captioning via deep convolutional neural network based encoder-decoder model

TL;DR

This paper introduces an improved Bengali image captioning system using a multimodal deep learning architecture that combines CNN-based sequence encoding with a pre-trained ResNet-50 image encoder, achieving state-of-the-art results.

Contribution

The paper presents a novel end-to-end Bengali image captioning model that outperforms existing methods by integrating CNN sequence encoding with a ResNet-50 image encoder.

Findings

Achieved new state-of-the-art scores on the BanglaLekhaImageCaptions dataset.

Demonstrated that the combined CNN and ResNet-50 approach improves caption accuracy.

Performed human evaluation confirming the qualitative improvements.

Abstract

Image Captioning is an arduous task of producing syntactically and semantically correct textual descriptions of an image in natural language with context related to the image. Existing notable pieces of research in Bengali Image Captioning (BIC) are based on encoder-decoder architecture. This paper presents an end-to-end image captioning system utilizing a multimodal architecture by combining a one-dimensional convolutional neural network (CNN) to encode sequence information with a pre-trained ResNet-50 model image encoder for extracting region-based visual features. We investigate our approach's performance on the BanglaLekhaImageCaptions dataset using the existing evaluation metrics and perform a human evaluation for qualitative analysis. Experiments show that our approach's language encoder captures the fine-grained information in the caption, and combined with the image features, it generates accurate and diversified caption. Our work outperforms all the existing BIC works and achieves a new state-of-the-art (SOTA) performance by scoring 0.651 on BLUE-1, 0.572 on CIDEr, 0.297 on METEOR, 0.434 on ROUGE, and 0.357 on SPICE.