VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context

TL;DR

This paper introduces VisionGraph, a benchmark for evaluating large multimodal models on complex visual graph theory problems, and proposes a Description-Program-Reasoning approach to improve reasoning accuracy, demonstrating GPT-4V's superior performance.

Contribution

The paper is the first to create a benchmark for multimodal graph theory problems and introduces a novel DPR method to enhance LMM reasoning capabilities.

Findings

GPT-4V outperforms Gemini Pro in multi-step reasoning

LMMs have limited perception accuracy for graphical structures

DPR significantly boosts reasoning performance, achieving SOTA results

Abstract

Large Multimodal Models (LMMs) have achieved impressive success in visual understanding and reasoning, remarkably improving the performance of mathematical reasoning in a visual context. Yet, a challenging type of visual math lies in the multimodal graph theory problem, which demands that LMMs understand the graphical structures accurately and perform multi-step reasoning on the visual graph. Additionally, exploring multimodal graph theory problems will lead to more effective strategies in fields like biology, transportation, and robotics planning. To step forward in this direction, we are the first to design a benchmark named VisionGraph, used to explore the capabilities of advanced LMMs in solving multimodal graph theory problems. It encompasses eight complex graph problem tasks, from connectivity to shortest path problems. Subsequently, we present a Description-Program-Reasoning (DPR) chain to enhance the logical accuracy of reasoning processes through graphical structure description generation and algorithm-aware multi-step reasoning. Our extensive study shows that 1) GPT-4V outperforms Gemini Pro in multi-step graph reasoning; 2) All LMMs exhibit inferior perception accuracy for graphical structures, whether in zero/few-shot settings or with supervised fine-tuning (SFT), which further affects problem-solving performance; 3) DPR significantly improves the multi-step graph reasoning capabilities of LMMs and the GPT-4V (DPR) agent achieves SOTA performance.