Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation

TL;DR

This paper introduces a novel evaluation method for multimodal large language models (MLLMs) that assesses their ability to detect, describe, and discriminate subtle visual differences between highly similar images, moving beyond traditional VQA metrics.

Contribution

It proposes a new self-retrieval based benchmark, D3, to evaluate fine-grained visual understanding in MLLMs, revealing current models' limitations in discriminating subtle visual differences.

Findings

Current models struggle with fine-grained visual discrimination.

Open-source models do not outperform random guessing on the benchmark.

The D3 benchmark enables whitebox evaluation of visual understanding.

Abstract

Visual Question Answering (VQA) with multiple choice questions enables a vision-centric evaluation of Multimodal Large Language Models (MLLMs). Although it reliably checks the existence of specific visual abilities, it is easier for the model to select an answer from multiple choices (VQA evaluation) than to generate the answer itself. In this work, we offer a novel perspective: we evaluate how well an MLLM understands a specific visual concept by its ability to uniquely describe two extremely similar images that differ only in the targeted visual concept. Specifically, we assess the ability of MLLMs to capture specific points of visual differences using self-retrieval, i.e., by retrieving the target image using its generated caption against the other image in the pair serving as the distractor. We curate 247 highly similar image pairs as part of the D3 benchmark. For each image pair, the model is prompted to: (1) Detect a specific visual difference, and (2) Describe the target image uniquely such that it (3) Discriminates the target image from the distractor. Self-retrieval within D3 enables whitebox evaluation across six different visual patterns, revealing that current models struggle to independently discern fine-grained visual differences, with open-source models failing to outperform random guess.