End-to-End Learning on Multimodal Knowledge Graphs

TL;DR

This paper introduces a multimodal message passing network that learns from both the structure and diverse node features of knowledge graphs, improving tasks like node classification and link prediction.

Contribution

It presents a novel end-to-end model that encodes multiple modalities of node features into a joint space, enhancing learning from heterogeneous knowledge graphs.

Findings

Multimodal information significantly improves performance in knowledge graph tasks.

The model effectively encodes five different types of node features.

Performance varies depending on data characteristics.

Abstract

Knowledge graphs enable data scientists to learn end-to-end on heterogeneous knowledge. However, most end-to-end models solely learn from the relational information encoded in graphs' structure: raw values, encoded as literal nodes, are either omitted completely or treated as regular nodes without consideration for their values. In either case we lose potentially relevant information which could have otherwise been exploited by our learning methods. We propose a multimodal message passing network which not only learns end-to-end from the structure of graphs, but also from their possibly divers set of multimodal node features. Our model uses dedicated (neural) encoders to naturally learn embeddings for node features belonging to five different types of modalities, including numbers, texts, dates, images and geometries, which are projected into a joint representation space together with their relational information. We implement and demonstrate our model on node classification and link prediction for artificial and real-worlds datasets, and evaluate the effect that each modality has on the overall performance in an inverse ablation study. Our results indicate that end-to-end multimodal learning from any arbitrary knowledge graph is indeed possible, and that including multimodal information can significantly affect performance, but that much depends on the characteristics of the data.