GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

TL;DR

GraphVLM introduces a comprehensive benchmark to evaluate vision-language models for structured multimodal graph reasoning, demonstrating their potential across various paradigms and datasets.

Contribution

This work systematically benchmarks VLMs for multimodal graph learning, exploring three integration paradigms and revealing the effectiveness of VLMs as a foundation for structured multimodal reasoning.

Findings

VLMs improve multimodal graph learning across multiple datasets.

VLM-as-Predictor yields the best performance among paradigms.

The benchmark code is publicly available for further research.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in aligning and understanding multimodal signals, yet their potential to reason over structured data, where multimodal entities are connected through explicit relational graphs, remains largely underexplored. Unlocking this capability is crucial for real-world applications such as social networks, recommendation systems, and scientific discovery, where multimodal information is inherently structured. To bridge this gap, we present GraphVLM, a systematic benchmark designed to evaluate and harness the capabilities of VLMs for multimodal graph learning (MMGL). GraphVLM investigates three complementary paradigms for integrating VLMs with graph reasoning: (1) VLM-as-Encoder, which enriches graph neural networks through multimodal feature fusion; (2) VLM-as-Aligner, which bridges modalities in latent or linguistic space to facilitate LLM-based structured reasoning; and (3) VLM-as-Predictor, which directly employs VLMs as multimodal backbones for graph learning tasks. Extensive experiments across six datasets from diverse domains demonstrate that VLMs enhance multimodal graph learning via all three roles. Among these paradigms, VLM-as-Predictor achieves the most substantial and consistent performance gains, revealing the untapped potential of vision-language models as a new foundation for multimodal graph learning. The benchmark code is publicly available at https://github.com/oamyjin/GraphVLM.