Towards Multimodal Graph Large Language Model

TL;DR

This paper proposes a unified framework for Multi-modal Graph Large Language Models (MG-LLM) to enhance generalization across diverse multi-modal graph data and tasks, emphasizing multi-granularity, multi-scale features, and natural language interaction.

Contribution

It introduces five key characteristics for MG-LLM, discusses challenges, reviews related work, and outlines future research directions for multi-modal graph learning.

Findings

Identifies five desired characteristics for MG-LLM

Highlights challenges and future directions in multi-modal graph learning

Summarizes relevant datasets for training MG-LLM

Abstract

Multi-modal graphs, which integrate diverse multi-modal features and relations, are ubiquitous in real-world applications. However, existing multi-modal graph learning methods are typically trained from scratch for specific graph data and tasks, failing to generalize across various multi-modal graph data and tasks. To bridge this gap, we explore the potential of Multi-modal Graph Large Language Models (MG-LLM) to unify and generalize across diverse multi-modal graph data and tasks. We propose a unified framework of multi-modal graph data, task, and model, discovering the inherent multi-granularity and multi-scale characteristics in multi-modal graphs. Specifically, we present five key desired characteristics for MG-LLM: 1) unified space for multi-modal structures and attributes, 2) capability of handling diverse multi-modal graph tasks, 3) multi-modal graph in-context learning, 4) multi-modal graph interaction with natural language, and 5) multi-modal graph reasoning. We then elaborate on the key challenges, review related works, and highlight promising future research directions towards realizing these ambitious characteristics. Finally, we summarize existing multi-modal graph datasets pertinent for model training. We believe this paper can contribute to the ongoing advancement of the research towards MG-LLM for generalization across multi-modal graph data and tasks.