AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs

TL;DR

This paper introduces AMVICC, a new benchmark for systematically profiling failure modes in vision-language models across image-to-text and text-to-image tasks, revealing shared and modality-specific limitations.

Contribution

The paper presents AMVICC, a novel benchmark that enables cross-modal failure mode analysis of VLMs and IGMs, highlighting their common and distinct weaknesses in visual reasoning.

Findings

Models share many failure modes across modalities.

IGMs struggle with manipulating visual components.

Failures are often model-specific and modality-specific.

Abstract

We investigated visual reasoning limitations of both multimodal large language models (MLLMs) and image generation models (IGMs) by creating a novel benchmark to systematically compare failure modes across image-to-text and text-to-image tasks, enabling cross-modal evaluation of visual understanding. Despite rapid growth in machine learning, vision language models (VLMs) still fail to understand or generate basic visual concepts such as object orientation, quantity, or spatial relationships, which highlighted gaps in elementary visual reasoning. By adapting MMVP benchmark questions into explicit and implicit prompts, we create \textit{AMVICC}, a novel benchmark for profiling failure modes across various modalities. After testing 11 MLLMs and 3 IGMs in nine categories of visual reasoning, our results show that failure modes are often shared between models and modalities, but certain failures are model-specific and modality-specific, and this can potentially be attributed to various factors. IGMs consistently struggled to manipulate specific visual components in response to prompts, especially in explicit prompts, suggesting poor control over fine-grained visual attributes. Our findings apply most directly to the evaluation of existing state-of-the-art models on structured visual reasoning tasks. This work lays the foundation for future cross-modal alignment studies, offering a framework to probe whether generation and interpretation failures stem from shared limitations to guide future improvements in unified vision-language modeling.