A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

TL;DR

This paper provides a comprehensive survey of prompt engineering techniques applied to vision-language models, covering methods, applications, and challenges across various model types to guide future research.

Contribution

It systematically reviews recent advances in prompt engineering for vision-language models, highlighting differences from language and vision models and discussing future research directions.

Findings

Summarizes prompting methods for multimodal-to-text, image-text matching, and text-to-image models.

Identifies key challenges and ethical issues in prompt engineering.

Outlines future research opportunities in the field.

Abstract

Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. Prompts can be created manually as natural language instructions or generated automatically as either natural language instructions or vector representations. Prompt engineering enables the ability to perform predictions based solely on prompts without updating model parameters, and the easier application of large pre-trained models in real-world tasks. In past years, Prompt engineering has been well-studied in natural language processing. Recently, it has also been intensively studied in vision-language modeling. However, there is currently a lack of a systematic overview of prompt engineering on pre-trained vision-language models. This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e.g. Flamingo), image-text matching models (e.g. CLIP), and text-to-image generation models (e.g. Stable Diffusion). For each type of model, a brief model summary, prompting methods, prompting-based applications, and the corresponding responsibility and integrity issues are summarized and discussed. Furthermore, the commonalities and differences between prompting on vision-language models, language models, and vision models are also discussed. The challenges, future directions, and research opportunities are summarized to foster future research on this topic.