Explainable Image Captioning using CNN- CNN architecture and Hierarchical Attention

TL;DR

This paper introduces an explainable image captioning model using a CNN-CNN architecture with hierarchical attention, enhancing transparency, speed, and accuracy in caption generation, and validated on the MSCOCO dataset.

Contribution

It proposes a novel CNN-based architecture with hierarchical attention for explainable image captioning, improving interpretability, speed, and accuracy over traditional black-box models.

Findings

Improved captioning accuracy demonstrated on MSCOCO dataset

Enhanced model interpretability through visualization of explanations

Faster caption generation compared to existing methods

Abstract

Image captioning is a technology that produces text-based descriptions for an image. Deep learning-based solutions built on top of feature recognition may very well serve the purpose. But as with any other machine learning solution, the user understanding in the process of caption generation is poor and the model does not provide any explanation for its predictions and hence the conventional methods are also referred to as Black-Box methods. Thus, an approach where the model's predictions are trusted by the user is needed to appreciate interoperability. Explainable AI is an approach where a conventional method is approached in a way that the model or the algorithm's predictions can be explainable and justifiable. Thus, this article tries to approach image captioning using Explainable AI such that the resulting captions generated by the model can be Explained and visualized. A newer architecture with a CNN decoder and hierarchical attention concept has been used to increase speed and accuracy of caption generation. Also, incorporating explainability to a model makes it more trustable when used in an application. The model is trained and evaluated using MSCOCO dataset and both quantitative and qualitative results are presented in this article.