Controllable Image Captioning via Prompting

TL;DR

This paper introduces a prompt-based method for controllable image captioning, enabling a single model to generate diverse, stylized captions across different domains without performance loss.

Contribution

It proposes a prompt learning framework with learnable vectors for flexible, multi-style caption generation in a unified model, surpassing heuristic prompt engineering.

Findings

Achieves controllable captioning with diverse styles.

Performs well on COCO and TextCaps benchmarks.

Maintains high performance across multiple domains.

Abstract

Despite the remarkable progress of image captioning, existing captioners typically lack the controllable capability to generate desired image captions, e.g., describing the image in a rough or detailed manner, in a factual or emotional view, etc. In this paper, we show that a unified model is qualified to perform well in diverse domains and freely switch among multiple styles. Such a controllable capability is achieved by embedding the prompt learning into the image captioning framework. To be specific, we design a set of prompts to fine-tune the pre-trained image captioner. These prompts allow the model to absorb stylized data from different domains for joint training, without performance degradation in each domain. Furthermore, we optimize the prompts with learnable vectors in the continuous word embedding space, avoiding the heuristic prompt engineering and meanwhile exhibiting superior performance. In the inference stage, our model is able to generate desired stylized captions by choosing the corresponding prompts. Extensive experiments verify the controllable capability of the proposed method. Notably, we achieve outstanding performance on two diverse image captioning benchmarks including COCO Karpathy split and TextCaps using a unified model.