MLaGA: Multimodal Large Language and Graph Assistant

TL;DR

MLaGA is a novel multimodal graph reasoning model that extends large language models to handle complex graphs with text and images, achieving superior performance in diverse tasks.

Contribution

Introduces MLaGA, a model that combines multimodal encoding and instruction tuning to enable LLMs to reason over multimodal graph data.

Findings

Outperforms baseline methods on multiple datasets.

Effective in supervised and transfer learning scenarios.

Successfully integrates multimodal features with graph structures.

Abstract

Large Language Models (LLMs) have demonstrated substantial efficacy in advancing graph-structured data analysis. Prevailing LLM-based graph methods excel in adapting LLMs to text-rich graphs, wherein node attributes are text descriptions. However, their applications to multimodal graphs--where nodes are associated with diverse attribute types, such as texts and images--remain underexplored, despite their ubiquity in real-world scenarios. To bridge the gap, we introduce the Multimodal Large Language and Graph Assistant (MLaGA), an innovative model that adeptly extends LLM capabilities to facilitate reasoning over complex graph structures and multimodal attributes. We first design a structure-aware multimodal encoder to align textual and visual attributes within a unified space through a joint graph pre-training objective. Subsequently, we implement a multimodal instruction-tuning approach to seamlessly integrate multimodal features and graph structures into the LLM through lightweight projectors. Extensive experiments across multiple datasets demonstrate the effectiveness of MLaGA compared to leading baseline methods, achieving superior performance in diverse graph learning tasks under both supervised and transfer learning scenarios.