GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs

TL;DR

GraphGPT-o is a novel framework that enhances multimodal comprehension and generation on attributed graphs by integrating semantic and structural information through hierarchical encoding and flexible inference strategies.

Contribution

It introduces a hierarchical aligner for deep graph encoding and explores inference methods for interleaved multimodal generation on graphs, advancing multimodal LLM capabilities.

Findings

Effective on three diverse datasets

Improves multimodal understanding and generation quality

Open-sourced datasets and code

Abstract

The rapid development of Multimodal Large Language Models (MLLMs) has enabled the integration of multiple modalities, including texts and images, within the large language model (LLM) framework. However, texts and images are usually interconnected, forming a multimodal attributed graph (MMAG). It is underexplored how MLLMs can incorporate the relational information (\textit{i.e.}, graph structure) and semantic information (\textit{i.e.,} texts and images) on such graphs for multimodal comprehension and generation. In this paper, we propose GraphGPT-o, which supports omni-multimodal understanding and creation on MMAGs. We first comprehensively study linearization variants to transform semantic and structural information as input for MLLMs. Then, we propose a hierarchical aligner that enables deep graph encoding, bridging the gap between MMAGs and MLLMs. Finally, we explore the inference choices, adapting MLLM to interleaved text and image generation in graph scenarios. Extensive experiments on three datasets from different domains demonstrate the effectiveness of our proposed method. Datasets and codes will be open-sourced upon acceptance.