Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignment

TL;DR

This paper introduces a context-adaptive multi-prompt embedding approach using large language models within CLIP to enhance semantic richness and diversity in vision-language alignment, leading to improved retrieval performance.

Contribution

It presents a novel multi-prompt embedding method with adaptive tokens and regularization techniques, advancing the semantic expressiveness of vision-language models.

Findings

Improves image-text retrieval accuracy

Enhances semantic diversity in embeddings

Achieves state-of-the-art results on benchmarks

Abstract

We propose Context-Adaptive Multi-Prompt Embedding, a novel approach to enrich semantic representations in vision-language contrastive learning. Unlike standard CLIP-style models that rely on a single text embedding, our method introduces multiple structured prompts, each containing a distinct adaptive token that captures diverse semantic aspects of the input text. We leverage a pretrained LLM as the text encoder within the CLIP framework, processing all prompts jointly in a single forward pass. The resulting prompt embeddings are combined into a unified text representation, enabling semantically richer alignment with visual features. To further promote semantic diversity and representation quality, we incorporate a diversity regularization loss and a negation-aware loss, encouraging specialization across prompts and improving contrastive discrimination. Our method achieves consistent improvements on both image-text and video-text retrieval benchmarks.