IMKGA-SM: Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling

TL;DR

This paper introduces IMKGA-SM, a novel interpretable model for multimodal knowledge graph link prediction that effectively combines multimodal data, models the task as a reinforcement learning problem, and outperforms state-of-the-art methods.

Contribution

The paper proposes a new multimodal link prediction model that integrates fine-grained fusion, reinforcement learning, and interpretability mechanisms, addressing challenges of complex data and sparse training.

Findings

IMKGA-SM outperforms SOTA baselines on multiple datasets.

Effective multimodal feature extraction using Vgg16 and OCR.

Novel sequence modeling approach enhances interpretability and accuracy.

Abstract

Multimodal knowledge graph link prediction aims to improve the accuracy and efficiency of link prediction tasks for multimodal data. However, for complex multimodal information and sparse training data, it is usually difficult to achieve interpretability and high accuracy simultaneously for most methods. To address this difficulty, a new model is developed in this paper, namely Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling (IMKGA-SM). First, a multi-modal fine-grained fusion method is proposed, and Vgg16 and Optical Character Recognition (OCR) techniques are adopted to effectively extract text information from images and images. Then, the knowledge graph link prediction task is modelled as an offline reinforcement learning Markov decision model, which is then abstracted into a unified sequence framework. An interactive perception-based reward expectation mechanism and a special causal masking mechanism are designed, which "converts" the query into an inference path. Then, an autoregressive dynamic gradient adjustment mechanism is proposed to alleviate the insufficient problem of multimodal optimization. Finally, two datasets are adopted for experiments, and the popular SOTA baselines are used for comparison. The results show that the developed IMKGA-SM achieves much better performance than SOTA baselines on multimodal link prediction datasets of different sizes.