Can Language Models Capture Graph Semantics? From Graphs to Language Model and Vice-Versa

TL;DR

This paper investigates whether Transformer-based deep learning models can effectively encode and reconstruct the full semantics of knowledge graphs, revealing limitations in their ability to preserve complex graph information.

Contribution

The study demonstrates that current Transformer models struggle to fully capture and reproduce the semantics of knowledge graphs due to structural differences.

Findings

Transformers cannot fully encode knowledge graph semantics

Disparity between graph structure and Transformer attention limits expressiveness

Knowledge graphs' directed and relationship-based info is not well-preserved

Abstract

Knowledge Graphs are a great resource to capture semantic knowledge in terms of entities and relationships between the entities. However, current deep learning models takes as input distributed representations or vectors. Thus, the graph is compressed in a vectorized representation. We conduct a study to examine if the deep learning model can compress a graph and then output the same graph with most of the semantics intact. Our experiments show that Transformer models are not able to express the full semantics of the input knowledge graph. We find that this is due to the disparity between the directed, relationship and type based information contained in a Knowledge Graph and the fully connected token-token undirected graphical interpretation of the Transformer Attention matrix.