Question-controlled Text-aware Image Captioning

TL;DR

This paper introduces a new task called Question-controlled Text-aware Image Captioning, which generates personalized captions based on questions about images with scene texts, and proposes a novel model that outperforms baselines.

Contribution

The paper defines a new challenging task, constructs datasets, and proposes a Geometry and Question Aware Model for personalized, question-guided image captioning involving scene texts.

Findings

GQAM outperforms baseline models on constructed datasets.

The model generates more informative and diverse captions.

The approach effectively integrates spatial and question-guided visual features.

Abstract

For an image with multiple scene texts, different people may be interested in different text information. Current text-aware image captioning models are not able to generate distinctive captions according to various information needs. To explore how to generate personalized text-aware captions, we define a new challenging task, namely Question-controlled Text-aware Image Captioning (Qc-TextCap). With questions as control signals, this task requires models to understand questions, find related scene texts and describe them together with objects fluently in human language. Based on two existing text-aware captioning datasets, we automatically construct two datasets, ControlTextCaps and ControlVizWiz to support the task. We propose a novel Geometry and Question Aware Model (GQAM). GQAM first applies a Geometry-informed Visual Encoder to fuse region-level object features and region-level scene text features with considering spatial relationships. Then, we design a Question-guided Encoder to select the most relevant visual features for each question. Finally, GQAM generates a personalized text-aware caption with a Multimodal Decoder. Our model achieves better captioning performance and question answering ability than carefully designed baselines on both two datasets. With questions as control signals, our model generates more informative and diverse captions than the state-of-the-art text-aware captioning model. Our code and datasets are publicly available at https://github.com/HAWLYQ/Qc-TextCap.