VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models

TL;DR

This paper introduces VRPTEST, a benchmark dataset for evaluating visual referring prompting strategies in large multimodal models, revealing significant impacts of prompt choice on model accuracy and understanding.

Contribution

It provides the first comprehensive analysis of visual referring prompting in LMMs, along with a new benchmark dataset and an automated evaluation framework.

Findings

Proprietary models outperform open-source models by 22.70% accuracy.

Prompt strategy significantly affects model accuracy, with variations from -17.5% to +7.3%.

Appropriate prompting improves context understanding, while poor prompts can cause answer rejection.

Abstract

With recent advancements in Large Multimodal Models (LMMs) across various domains, a novel prompting method called visual referring prompting has emerged, showing significant potential in enhancing human-computer interaction within multimodal systems. This method offers a more natural and flexible approach to human interaction with these systems compared to traditional text descriptions or coordinates. However, the categorization of visual referring prompting remains undefined, and its impact on the performance of LMMs has yet to be formally examined. In this study, we conduct the first comprehensive analysis of LMMs using a variety of visual referring prompting strategies. We introduce a benchmark dataset called VRPTEST, comprising 3 different visual tasks and 2,275 images, spanning diverse combinations of prompt strategies. Using VRPTEST, we conduct a comprehensive evaluation of eight versions of prominent open-source and proprietary foundation models, including two early versions of GPT-4V. We develop an automated assessment framework based on software metamorphic testing techniques to evaluate the accuracy of LMMs without the need for human intervention or manual labeling. We find that the current proprietary models generally outperform the open-source ones, showing an average accuracy improvement of 22.70%; however, there is still potential for improvement. Moreover, our quantitative analysis shows that the choice of prompt strategy significantly affects the accuracy of LMMs, with variations ranging from -17.5% to +7.3%. Further case studies indicate that an appropriate visual referring prompting strategy can improve LMMs' understanding of context and location information, while an unsuitable one might lead to answer rejection. We also provide insights on minimizing the negative impact of visual referring prompting on LMMs.