Multimodal Prompt Retrieval for Generative Visual Question Answering

TL;DR

This paper introduces a multimodal prompt retrieval approach for generative visual question answering, improving zero-shot adaptation and open-set answer generation, especially in low-resource domains like medicine.

Contribution

It proposes a novel generative VQA model with multimodal prompt retrieval that enhances domain adaptation and open-set answer generation.

Findings

Outperforms non-retrieval models by up to 30% accuracy in medical VQA.

Enables rapid zero-shot adaptation to unseen data.

Improves generalization in low-resource and domain-shift scenarios.

Abstract

Recent years have witnessed impressive results of pre-trained vision-language models on knowledge-intensive tasks such as visual question answering (VQA). Despite the recent advances in VQA, existing methods mainly adopt a discriminative formulation that predicts answers within a pre-defined label set, leading to easy overfitting on low-resource domains with limited labeled data (e.g., medicine) and poor generalization under domain shift to another dataset. To tackle this limitation, we propose a novel generative model enhanced by multimodal prompt retrieval (MPR) that integrates retrieved prompts and multimodal features to generate answers in free text. Our generative model enables rapid zero-shot dataset adaptation to unseen data distributions and open-set answer labels across datasets. Our experiments on medical VQA tasks show that MPR outperforms its non-retrieval counterpart by up to 30% accuracy points in a few-shot domain adaptation setting.