PromptCap: Prompt-Guided Task-Aware Image Captioning

TL;DR

PromptCap is a prompt-guided image captioning model that generates task-aware captions to improve knowledge-based visual question answering, outperforming generic captioning methods and achieving state-of-the-art results.

Contribution

It introduces a novel prompt-guided captioning approach that enhances image descriptions for better VQA performance, trained with GPT-3 synthesized data.

Findings

Significantly outperforms generic captions on VQA tasks

Achieves state-of-the-art accuracy on OK-VQA and A-OKVQA datasets

Generalizes well to unseen domains in zero-shot settings

Abstract

Knowledge-based visual question answering (VQA) involves questions that require world knowledge beyond the image to yield the correct answer. Large language models (LMs) like GPT-3 are particularly helpful for this task because of their strong knowledge retrieval and reasoning capabilities. To enable LM to understand images, prior work uses a captioning model to convert images into text. However, when summarizing an image in a single caption sentence, which visual entities to describe are often underspecified. Generic image captions often miss visual details essential for the LM to answer visual questions correctly. To address this challenge, we propose PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs. Different from generic captions, PromptCap takes a natural-language prompt to control the visual entities to describe in the generated caption. The prompt contains a question that the caption should aid in answering. To avoid extra annotation, PromptCap is trained by examples synthesized with GPT-3 and existing datasets. We demonstrate PromptCap's effectiveness on an existing pipeline in which GPT-3 is prompted with image captions to carry out VQA. PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60.4% on OK-VQA and 59.6% on A-OKVQA). Zero-shot results on WebQA show that PromptCap generalizes well to unseen domains.