AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs

TL;DR

AutoV introduces a loss-based prompt retrieval framework that automatically identifies the most suitable visual prompts for large vision-language models, significantly improving their performance across multiple tasks without manual prompt annotation.

Contribution

It proposes a novel loss-oriented ranking method for automatic visual prompt retrieval, addressing the limitations of prompt engineering in LVLMs.

Findings

AutoV improves LLaVA-OV performance by 10.2% on VizWiz.

AutoV boosts Qwen2.5-VL accuracy by 3.8% on MMMU.

The framework enhances various LVLM tasks including image understanding, captioning, grounding, and classification.

Abstract

Inspired by text prompts in large language models, visual prompts have been explored to enhance the perceptual capabilities of large vision-language models (LVLMs). However, performance tends to saturate under single visual prompt designs, making further prompt engineering increasingly ineffective. To address this limitation, we shift from prompt engineering to prompt retrieval and propose AutoV, a lightweight framework for instance-adaptive visual prompt identification. Given an input image and a textual query, AutoV automatically locates the most suitable visual prompt from a diverse candidate pool. Training such a retrieval framework requires prompt-level supervision, yet prompt quality is inherently ambiguous and difficult to assess reliably, even for humans. To enable automatic supervision, we evaluate visual prompts using a pre-trained LVLM and label them according to their prediction losses. Using the loss-oriented ranking as a robust training signal, AutoV learns to retrieve the query-aware optimal prompt for each instance without manual annotation. Experiments indicate that AutoV enhances the performance of various LVLMs on image understanding, captioning, grounding, and classification tasks. For example, AutoV improves LLaVA-OV by $\textbf{10.2}\%$ on VizWiz and boosts Qwen2.5-VL by $\textbf{3.8}\%$ on MMMU, respectively.