HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs

TL;DR

HERGC introduces a novel framework combining heterogeneous expert retrieval and generative large language models to improve multimodal knowledge graph completion, outperforming existing methods on standard benchmarks.

Contribution

HERGC is the first to integrate heterogeneous expert retrieval with generative LLMs for MMKG completion, enhancing reasoning and handling multimodal data.

Findings

HERGC achieves state-of-the-art results on three MMKG benchmarks.

The framework effectively fuses multimodal information for better inference.

HERGC demonstrates robustness across diverse datasets.

Abstract

Multimodal knowledge graphs (MMKGs) enrich traditional knowledge graphs (KGs) by incorporating diverse modalities such as images and text. multimodal knowledge graph completion (MMKGC) seeks to exploit these heterogeneous signals to infer missing facts, thereby mitigating the intrinsic incompleteness of MMKGs. Existing MMKGC methods typically leverage only the information contained in the MMKGs under the closed-world assumption and adopt discriminative training objectives, which limits their reasoning capacity during completion. Recent large language models (LLMs), empowered by massive parameter scales and pretraining on vast corpora, have demonstrated strong reasoning abilities across various tasks. However, their potential in MMKGC remains largely unexplored. To bridge this gap, we propose HERGC, a flexible Heterogeneous Experts Representation and Generative Completion framework for MMKGs. HERGC first deploys a Heterogeneous Experts Representation Retriever that enriches and fuses multimodal information and retrieves a compact candidate set for each incomplete triple. It then uses a Generative LLM Predictor, implemented via either in-context learning or lightweight fine-tuning, to accurately identify the correct answer from these candidates. Extensive experiments on three standard MMKG benchmarks demonstrate HERGC's effectiveness and robustness, achieving superior performance over existing methods.