VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings

TL;DR

VL-KGE leverages vision-language models to create unified, multimodal embeddings for knowledge graphs, significantly improving link prediction by better aligning diverse modalities.

Contribution

This paper introduces VL-KGE, a novel framework that combines VLMs with structured relational modeling for enhanced multimodal knowledge graph embeddings.

Findings

VL-KGE outperforms traditional KGE methods in link prediction.

Experiments on WN9-IMG and new art MKGs show improved multimodal reasoning.

VLMs effectively align diverse modalities within knowledge graphs.

Abstract

Real-world multimodal knowledge graphs (MKGs) are inherently heterogeneous, modeling entities that are associated with diverse modalities. Traditional knowledge graph embedding (KGE) methods excel at learning continuous representations of entities and relations, yet they are typically designed for unimodal settings. Recent approaches extend KGE to multimodal settings but remain constrained, often processing modalities in isolation, resulting in weak cross-modal alignment, and relying on simplistic assumptions such as uniform modality availability across entities. Vision-Language Models (VLMs) offer a powerful way to align diverse modalities within a shared embedding space. We propose Vision-Language Knowledge Graph Embeddings (VL-KGE), a framework that integrates cross-modal alignment from VLMs with structured relational modeling to learn unified multimodal representations of knowledge graphs. Experiments on WN9-IMG and two novel fine art MKGs, WikiArt-MKG-v1 and WikiArt-MKG-v2, demonstrate that VL-KGE consistently improves over traditional unimodal and multimodal KGE methods in link prediction tasks. Our results highlight the value of VLMs for multimodal KGE, enabling more robust and structured reasoning over large-scale heterogeneous knowledge graphs.