AIC-AB NET: A Neural Network for Image Captioning with Spatial Attention and Text Attributes

TL;DR

This paper introduces AIC-AB NET, a neural network that enhances image captioning by integrating spatial attention and text attributes, demonstrating superior performance on MS COCO and a new Fashion dataset.

Contribution

The novel AIC-AB NET combines spatial attention with text attributes in an encoder-decoder framework for improved image captioning.

Findings

Outperforms baseline adaptive attention network by 0.017 CIDEr score on MS COCO

Achieves 0.095 CIDEr score improvement on Fashion dataset

Effective in single-object image captioning scenarios

Abstract

Image captioning is a significant field across computer vision and natural language processing. We propose and present AIC-AB NET, a novel Attribute-Information-Combined Attention-Based Network that combines spatial attention architecture and text attributes in an encoder-decoder. For caption generation, adaptive spatial attention determines which image region best represents the image and whether to attend to the visual features or the visual sentinel. Text attribute information is synchronously fed into the decoder to help image recognition and reduce uncertainty. We have tested and evaluated our AICAB NET on the MS COCO dataset and a new proposed Fashion dataset. The Fashion dataset is employed as a benchmark of single-object images. The results show the superior performance of the proposed model compared to the state-of-the-art baseline and ablated models on both the images from MSCOCO and our single-object images. Our AIC-AB NET outperforms the baseline adaptive attention network by 0.017 (CIDEr score) on the MS COCO dataset and 0.095 (CIDEr score) on the Fashion dataset.