Multi-Modal Image Captioning for the Visually Impaired

TL;DR

This paper enhances image captioning for the visually impaired by integrating detected textual information into the captioning process, significantly improving performance on a benchmark dataset.

Contribution

It introduces a modified AoANet model that leverages image text detection and a pointer-generator mechanism to improve caption accuracy for scenes containing text.

Findings

35% improvement in CIDEr score

16.2% improvement in SPICE score

Outperforms baseline AoANet on VizWiz dataset

Abstract

One of the ways blind people understand their surroundings is by clicking images and relying on descriptions generated by image captioning systems. Current work on captioning images for the visually impaired do not use the textual data present in the image when generating captions. This problem is critical as many visual scenes contain text. Moreover, up to 21% of the questions asked by blind people about the images they click pertain to the text present in them. In this work, we propose altering AoANet, a state-of-the-art image captioning model, to leverage the text detected in the image as an input feature. In addition, we use a pointer-generator mechanism to copy the detected text to the caption when tokens need to be reproduced accurately. Our model outperforms AoANet on the benchmark dataset VizWiz, giving a 35% and 16.2% performance improvement on CIDEr and SPICE scores, respectively.