Noise-powered Multi-modal Knowledge Graph Representation Framework

TL;DR

This paper introduces SNAG, a Transformer-based framework with noise masking for multi-modal knowledge graph embedding, achieving state-of-the-art results in completion and alignment tasks across multiple datasets.

Contribution

The paper presents a novel noise-powered Transformer model, SNAG, for robust multi-modal knowledge graph embedding, improving performance on key tasks and datasets.

Findings

SNAG achieves state-of-the-art results on 10 datasets.

SNAG improves multi-modal knowledge graph completion and entity alignment.

SNAG enhances existing methods with stable performance gains.

Abstract

The rise of Multi-modal Pre-training highlights the necessity for a unified Multi-Modal Knowledge Graph (MMKG) representation learning framework. Such a framework is essential for embedding structured knowledge into multi-modal Large Language Models effectively, alleviating issues like knowledge misconceptions and multi-modal hallucinations. In this work, we explore the efficacy of models in accurately embedding entities within MMKGs through two pivotal tasks: Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA). Building on this foundation, we propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking to robustly integrate multi-modal entity features in KGs. By incorporating specific training objectives for both MKGC and MMEA, our approach achieves SOTA performance across a total of ten datasets, demonstrating its versatility. Moreover, SNAG can not only function as a standalone model but also enhance other existing methods, providing stable performance improvements. Code and data are available at https://github.com/zjukg/SNAG.