VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models

TL;DR

VP-Bench introduces a comprehensive benchmark to evaluate multimodal large language models' ability to perceive and utilize visual prompts, addressing a key gap in understanding their performance in grounded vision-language tasks.

Contribution

The paper presents VP-Bench, a two-stage evaluation framework for assessing MLLMs' VP perception and application, including a large-scale dataset and analysis of influencing factors.

Findings

28 MLLMs evaluated, including GPT-4o and open-source models.

VP understanding varies significantly with prompt attributes and model scale.

VP-Bench sets a new standard for grounded referring question comprehension.

Abstract

Multimodal large language models (MLLMs) have enabled a wide range of advanced vision-language applications, including fine-grained object recognition and contextual understanding. When querying specific regions or objects in an image, human users naturally use "visual prompts" (VPs), such as bounding boxes, to provide reference. However, no existing benchmark systematically evaluates the ability of MLLMs to interpret such VPs. This gap leaves it unclear whether current MLLMs can effectively recognize VPs, an intuitive prompting method for humans, and use them to solve problems. To address this limitation, we introduce VP-Bench, a benchmark for assessing MLLMs' capability in VP perception and utilization. VP-Bench employs a two-stage evaluation framework: Stage 1 examines models' ability to perceive VPs in natural scenes, using 30k visualized prompts spanning eight shapes and 355 attribute combinations. Stage 2 investigates the impact of VPs on downstream tasks, measuring their effectiveness in real-world problem-solving scenarios. Using VP-Bench, we evaluate 28 MLLMs, including proprietary systems (e.g., GPT-4o) and open-source models (e.g., InternVL3 and Qwen2.5-VL), and provide a comprehensive analysis of factors that affect VP understanding, such as variations in VP attributes, question arrangement, and model scale. VP-Bench establishes a new reference framework for studying how MLLMs comprehend and resolve grounded referring questions.