Oedipus and the Sphinx: Benchmarking and Improving Visual Language Models for Complex Graphic Reasoning

TL;DR

This paper introduces ReasonBench, a comprehensive benchmark for evaluating visual language models on complex graphic reasoning tasks, revealing current limitations and proposing strategies to significantly improve their reasoning capabilities.

Contribution

It presents the first structured benchmark for complex graphic reasoning in VLMs and introduces dual optimization strategies that enhance model performance and interpretability.

Findings

VLMs show significant limitations in complex graphic reasoning.

The proposed strategies improve VLM performance by 33.5%.

ReasonBench covers diverse reasoning dimensions and real-world questions.

Abstract

Evaluating the performance of visual language models (VLMs) in graphic reasoning tasks has become an important research topic. However, VLMs still show obvious deficiencies in simulating human-level graphic reasoning capabilities, especially in complex graphic reasoning and abstract problem solving, which are less studied and existing studies only focus on simple graphics. To evaluate the performance of VLMs in complex graphic reasoning, we propose ReasonBench, the first evaluation benchmark focused on structured graphic reasoning tasks, which includes 1,613 questions from real-world intelligence tests. ReasonBench covers reasoning dimensions related to location, attribute, quantity, and multi-element tasks, providing a comprehensive evaluation of the performance of VLMs in spatial, relational, and abstract reasoning capabilities. We benchmark 11 mainstream VLMs (including closed-source and open-source models) and reveal significant limitations of current models. Based on these findings, we propose a dual optimization strategy: Diagrammatic Reasoning Chain (DiaCoT) enhances the interpretability of reasoning by decomposing layers, and ReasonTune enhances the task adaptability of model reasoning through training, all of which improves VLM performance by 33.5\%. All experimental data and code are in the repository: https://huggingface.co/datasets/cistine/ReasonBench.