DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning

TL;DR

DiffCap-Bench introduces a diverse, challenging benchmark for image difference captioning, emphasizing semantic accuracy and reasoning, with an evaluation protocol that aligns well with human judgments and downstream tasks.

Contribution

This work presents a comprehensive IDC benchmark with an LLM-based evaluation protocol, addressing diversity, complexity, and evaluation limitations of prior benchmarks.

Findings

Significant performance gaps between proprietary and open-source models.

Reasoning capability is critical for accurate difference captioning.

Strong correlation between benchmark results and downstream image editing quality.

Abstract

Image Difference Captioning (IDC) generates natural language descriptions that precisely identify differences between two images, serving as a key benchmark for fine-grained change perception, cross-modal reasoning, and image editing data construction. However, existing benchmarks lack diversity and compositional complexity, and standard lexical-overlap metrics (e.g., BLEU, METEOR) fail to capture semantic consistency or penalize hallucinations, which together prevent a comprehensive and robust evaluation of multimodal large language models (MLLMs) on IDC. To address these gaps, we introduce DiffCap-Bench, a comprehensive IDC benchmark covering ten distinct difference categories to ensure diversity and compositional complexity. Furthermore, we propose an LLM-as-a-Judge evaluation protocol grounded in human-validated Difference Lists, enabling a robust assessment of models' ability to both capture and describe visual changes. Through extensive evaluation of state-of-the-art MLLMs, we reveal significant performance gaps between proprietary and open-source models, highlight the critical importance of reasoning capability, and identify clear limitations in model scaling. Our framework also demonstrates strong alignment with human expert judgments and strong correlation with downstream image editing data construction quality. These findings establish DiffCap-Bench as both a reliable IDC evaluation framework and a practical predictor of downstream utility. The benchmark and code will be made publicly available to support further research.