MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion

TL;DR

MoSE introduces a modality-split approach and ensemble inference for multimodal knowledge graph completion, effectively reducing modality interference and dynamically modeling modality importance, leading to improved prediction accuracy.

Contribution

The paper proposes a novel modality-split relation embedding and ensemble inference framework for MKGC, addressing interference and importance modeling issues.

Findings

MoSE outperforms existing MKGC methods on three datasets.

Modality-split embeddings reduce interference between modalities.

Ensemble inference effectively models modality importance dynamically.

Abstract

Multimodal knowledge graph completion (MKGC) aims to predict missing entities in MKGs. Previous works usually share relation representation across modalities. This results in mutual interference between modalities during training, since for a pair of entities, the relation from one modality probably contradicts that from another modality. Furthermore, making a unified prediction based on the shared relation representation treats the input in different modalities equally, while their importance to the MKGC task should be different. In this paper, we propose MoSE, a Modality Split representation learning and Ensemble inference framework for MKGC. Specifically, in the training phase, we learn modality-split relation embeddings for each modality instead of a single modality-shared one, which alleviates the modality interference. Based on these embeddings, in the inference phase, we first make modality-split predictions and then exploit various ensemble methods to combine the predictions with different weights, which models the modality importance dynamically. Experimental results on three KG datasets show that MoSE outperforms state-of-the-art MKGC methods. Codes are available at https://github.com/OreOZhao/MoSE4MKGC.