PairBench: Are Vision-Language Models Reliable at Comparing What They See?

TL;DR

PairBench provides a comprehensive framework to evaluate vision-language models' ability to compare images reliably, highlighting their strengths, weaknesses, and correlation with existing benchmarks.

Contribution

Introduces PairBench, a systematic and customizable evaluation framework for assessing VLMs' image comparison capabilities across multiple metrics.

Findings

No model excels across all metrics

VLMs often fail to maintain symmetric similarity scores

Performance correlates with existing benchmarks

Abstract

Understanding how effectively large vision language models (VLMs) compare visual inputs is crucial across numerous applications, yet this fundamental capability remains insufficiently assessed. While VLMs are increasingly deployed for tasks requiring comparative judgment, including automated evaluation, re-ranking, and retrieval-augmented generation, no systematic framework exists to measure their performance in these scenarios. We present PairBench, a simple framework that evaluates VLMs as customizable similarity tools using widely available image datasets. Our approach introduces four key metrics for reliable comparison: alignment with human annotations, consistency across pair ordering, distribution smoothness, and controllability through prompting. Our analysis reveals that no model consistently excels across all metrics, with each demonstrating distinct strengths and weaknesses. Most concerning is the widespread inability of VLMs to maintain symmetric similarity scores. Interestingly, we demonstrate that performance on our benchmark strongly correlates with popular benchmarks used for more complex tasks, while providing additional metrics into controllability, smoothness and ordering. This makes PairBench a unique and comprehensive framework to evaluate the performance of VLMs for automatic evaluation depending on the task.