JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images

TL;DR

JourneyBench is a new, challenging benchmark for vision-language understanding that tests models' fine-grained reasoning on generated images in unusual scenarios, revealing limitations in current multimodal models.

Contribution

The paper introduces JourneyBench, a comprehensive human-annotated benchmark specifically designed to evaluate fine-grained multimodal reasoning in generated images, addressing limitations of existing benchmarks.

Findings

State-of-the-art models perform poorly on JourneyBench.

JourneyBench reveals gaps in models' visual reasoning abilities.

Benchmark is highly challenging, exposing weaknesses in current models.

Abstract

Existing vision-language understanding benchmarks largely consist of images of objects in their usual contexts. As a consequence, recent multimodal large language models can perform well with only a shallow visual understanding by relying on background language biases. Thus, strong performance on these benchmarks does not necessarily correlate with strong visual understanding. In this paper, we release JourneyBench, a comprehensive human-annotated benchmark of generated images designed to assess the model's fine-grained multimodal reasoning abilities across five tasks: complementary multimodal chain of thought, multi-image VQA, imaginary image captioning, VQA with hallucination triggers, and fine-grained retrieval with sample-specific distractors. Unlike existing benchmarks, JourneyBench explicitly requires fine-grained multimodal reasoning in unusual imaginary scenarios where language bias and holistic image gist are insufficient. We benchmark state-of-the-art models on JourneyBench and analyze performance along a number of fine-grained dimensions. Results across all five tasks show that JourneyBench is exceptionally challenging for even the best models, indicating that models' visual reasoning abilities are not as strong as they first appear. We discuss the implications of our findings and propose avenues for further research.