HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning

TL;DR

This paper introduces HyperGVL, a comprehensive benchmark for evaluating and enhancing large vision-language models' ability to understand and reason with hypergraphs across diverse tasks and real-world applications.

Contribution

It presents the first benchmark for hypergraph understanding in LVLMs, evaluates 12 models on 84,000 QA samples, and proposes a novel adaptive routing method to improve hypergraph reasoning.

Findings

HyperGVL effectively assesses LVLMs' hypergraph reasoning capabilities.

The proposed WiseHyGR router enhances model performance in hypergraph tasks.

Evaluation reveals strengths and limitations of current LVLMs in hypergraph understanding.

Abstract

Large Vision-Language Models (LVLMs) consistently require new arenas to guide their expanding boundaries, yet their capabilities with hypergraphs remain unexplored. In the real world, hypergraphs have significant practical applications in areas such as life sciences and social communities. Recent advancements in LVLMs have shown promise in understanding complex topologies, yet there remains a lack of a benchmark to delineate the capabilities of LVLMs with hypergraphs, leaving the boundaries of their abilities unclear. To fill this gap, in this paper, we introduce $\texttt{HyperGVL}$, the first benchmark to evaluate the proficiency of LVLMs in hypergraph understanding and reasoning. $\texttt{HyperGVL}$ provides a comprehensive assessment of 12 advanced LVLMs across 84,000 vision-language question-answering (QA) samples spanning 12 tasks, ranging from basic component counting to complex NP-hard problem reasoning. The involved hypergraphs contain multiscale synthetic structures and real-world citation and protein networks. Moreover, we examine the effects of 12 textual and visual hypergraph representations and introduce a generalizable router $\texttt{WiseHyGR}$ that improves LVLMs in hypergraph via learning adaptive representations. We believe that this work is a step forward in connecting hypergraphs with LVLMs.