Understanding Guided Image Captioning Performance across Domains

TL;DR

This paper introduces a guided image captioning method that uses guiding text to focus on specific concepts, demonstrating better out-of-domain generalization and emphasizing the importance of diverse training data for real-world applications.

Contribution

The authors propose a Transformer-based multimodal model for guided image captioning that effectively incorporates guiding text and shows improved out-of-domain performance.

Findings

Guided captioning models trained on Conceptual Captions generalize better out-of-domain.

Increased style diversity enhances captioning performance in wild settings.

Large, unrestricted datasets are crucial for effective in-the-wild guided captioning.

Abstract

Image captioning models generally lack the capability to take into account user interest, and usually default to global descriptions that try to balance readability, informativeness, and information overload. On the other hand, VQA models generally lack the ability to provide long descriptive answers, while expecting the textual question to be quite precise. We present a method to control the concepts that an image caption should focus on, using an additional input called the guiding text that refers to either groundable or ungroundable concepts in the image. Our model consists of a Transformer-based multimodal encoder that uses the guiding text together with global and object-level image features to derive early-fusion representations used to generate the guided caption. While models trained on Visual Genome data have an in-domain advantage of fitting well when guided with automatic object labels, we find that guided captioning models trained on Conceptual Captions generalize better on out-of-domain images and guiding texts. Our human-evaluation results indicate that attempting in-the-wild guided image captioning requires access to large, unrestricted-domain training datasets, and that increased style diversity (even without increasing the number of unique tokens) is a key factor for improved performance.