HittER: Hierarchical Transformers for Knowledge Graph Embeddings

TL;DR

HittER introduces a hierarchical Transformer model for knowledge graph embeddings that jointly captures entity-relation composition and relational context, achieving state-of-the-art results in link prediction and question answering tasks.

Contribution

The paper presents a novel hierarchical Transformer architecture for knowledge graph embedding that effectively models local neighborhood information and relational context.

Findings

HittER outperforms existing models on multiple link prediction datasets.

HittER integrated into BERT improves performance on Freebase question answering.

The hierarchical Transformer captures complex relational structures effectively.

Abstract

This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity's neighborhood. Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational context and the source entity itself. Experimental results show that HittER achieves new state-of-the-art results on multiple link prediction datasets. We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.