NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision

TL;DR

NPHardEval4V introduces a new multimodal benchmark based on NP-hard problems to evaluate the reasoning capabilities of large vision-language models, revealing their limitations in complex combinatorial tasks.

Contribution

This work presents NPHardEval4V, a novel benchmark suite that assesses LVLMs on structured, logic-driven problems combining visual and linguistic reasoning, filling a gap in existing evaluation methods.

Findings

Models perform well on perception but struggle with optimization and abstraction.

No single model shows consistent reasoning across all NP-hard tasks.

Current architectures have fundamental limitations in complex combinatorial reasoning.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in multimodal understanding, yet their reasoning abilities remain underexplored. Existing benchmarks tend to focus on perception or text-based comprehension, offering limited insight into how well these models perform on structured, logic-driven tasks that require both visual and linguistic reasoning. To address this gap, we introduce NPHardEval4V, a multimodal benchmark suite grounded in four classical NP-hard problems: Knapsack, Set Cover, Traveling Salesperson, and Vertex Cover. Each task is presented through a combination of structured visual layouts and textual prompts, designed to assess the ability of LVLMs to perform combinatorial reasoning under visual-linguistic constraints. We evaluate a set of advanced open-source and closed-source vision-language models under a unified prompting and problem representation framework. This enables fair comparison across models and task types, while isolating key variables affecting performance. Our results show that while these models perform reasonably well on perception-based inputs, they struggle with global optimization, abstraction, and constraint satisfaction. No single model demonstrates consistent reasoning capability across all problem types, and common failure patterns reveal fundamental limitations in current architectures. By leveraging the structure and complexity of NP-hard problems, NPHardEval4V provides a scalable, interpretable, and challenging testbed for diagnosing reasoning behaviors in LVLMs. We hope this benchmark can support the community in building more robust, inference-capable multimodal systems. The benchmark dataset and code are available at https://github.com/lizhouf/NPHardEval4.