Visual Adaptive Prompting for Compositional Zero-Shot Learning

TL;DR

This paper introduces a Visual Adaptive Prompting System (VAPS) that dynamically retrieves visual prompts based on image features, significantly improving compositional zero-shot learning performance in vision-language models.

Contribution

The paper proposes a novel visual prompt retrieval mechanism and a visual prompt adapter, enhancing the generalization of VLMs for CZSL tasks through adaptive visual prompts.

Findings

Achieves state-of-the-art results on three CZSL benchmarks.

Effective in both closed and open-world scenarios.

Demonstrates improved generalization over static prompting methods.

Abstract

Vision-Language Models (VLMs) have demonstrated impressive multimodal capabilities in learning joint representations of visual and textual data, making them powerful tools for tasks such as Compositional Zero-Shot Learning (CZSL). CZSL requires models to generalize to novel combinations of visual primitives--such as attributes and objects--that were not explicitly encountered during training. Recent works in prompting for CZSL have focused on modifying inputs for the text encoder, often using static prompts that do not change across varying visual contexts. However, these approaches struggle to fully capture varying visual contexts, as they focus on text adaptation rather than leveraging visual features for compositional reasoning. To address this, we propose a Visual Adaptive Prompting System (VAPS) that leverages a learnable visual prompt repository and similarity-based retrieval mechanism within the framework of VLMs to bridge the gap between semantic and visual features. Our method introduces a dynamic visual prompt repository mechanism that selects the most relevant attribute and object prompts based on the visual features of the image. Our proposed system includes a visual prompt adapter that encourages the model to learn a more generalizable embedding space. Experiments on three CZSL benchmarks, across both closed and open-world scenarios, demonstrate state-of-the-art results.