MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs

TL;DR

This paper introduces MLLM-CompBench, a comprehensive benchmark for evaluating the comparative reasoning abilities of multimodal large language models across diverse visual domains and dimensions.

Contribution

The paper presents a new benchmark dataset with 40K image pairs and questions to assess MLLMs' comparative reasoning, highlighting current limitations and guiding future improvements.

Findings

Recent MLLMs show significant shortcomings in comparative reasoning.

The benchmark covers eight dimensions of comparison across diverse visual domains.

Evaluation results identify specific areas for model enhancement.

Abstract

The ability to compare objects, scenes, or situations is crucial for effective decision-making and problem-solving in everyday life. For instance, comparing the freshness of apples enables better choices during grocery shopping while comparing sofa designs helps optimize the aesthetics of our living space. Despite its significance, the comparative capability is largely unexplored in artificial general intelligence (AGI). In this paper, we introduce MLLM-CompBench, a benchmark designed to evaluate the comparative reasoning capability of multimodal large language models (MLLMs). MLLM-CompBench mines and pairs images through visually oriented questions covering eight dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. We curate a collection of around 40K image pairs using metadata from diverse vision datasets and CLIP similarity scores. These image pairs span a broad array of visual domains, including animals, fashion, sports, and both outdoor and indoor scenes. The questions are carefully crafted to discern relative characteristics between two images and are labeled by human annotators for accuracy and relevance. We use MLLM-CompBench to evaluate recent MLLMs, including GPT-4V(ision), Gemini-Pro, and LLaVA-1.6. Our results reveal notable shortcomings in their comparative abilities. We believe MLLM-COMPBENCH not only sheds light on these limitations but also establishes a solid foundation for future enhancements in the comparative capability of MLLMs.