Transformer-based Reasoning for Learning Evolutionary Chain of Events on Temporal Knowledge Graph

TL;DR

This paper introduces ECEformer, a Transformer-based model that effectively learns the evolution of events in temporal knowledge graphs by modeling intra- and inter-quadruple relationships, achieving state-of-the-art results.

Contribution

The paper proposes a novel Transformer-based approach that models the evolutionary chain of events in TKGs, addressing limitations of previous methods in capturing internal structure and temporal correlations.

Findings

Achieves state-of-the-art performance on six benchmark datasets.

Effectively models the evolution of events in temporal knowledge graphs.

Enhances temporal reasoning with a time prediction task.

Abstract

Temporal Knowledge Graph (TKG) reasoning often involves completing missing factual elements along the timeline. Although existing methods can learn good embeddings for each factual element in quadruples by integrating temporal information, they often fail to infer the evolution of temporal facts. This is mainly because of (1) insufficiently exploring the internal structure and semantic relationships within individual quadruples and (2) inadequately learning a unified representation of the contextual and temporal correlations among different quadruples. To overcome these limitations, we propose a novel Transformer-based reasoning model (dubbed ECEformer) for TKG to learn the Evolutionary Chain of Events (ECE). Specifically, we unfold the neighborhood subgraph of an entity node in chronological order, forming an evolutionary chain of events as the input for our model. Subsequently, we utilize a Transformer encoder to learn the embeddings of intra-quadruples for ECE. We then craft a mixed-context reasoning module based on the multi-layer perceptron (MLP) to learn the unified representations of inter-quadruples for ECE while accomplishing temporal knowledge reasoning. In addition, to enhance the timeliness of the events, we devise an additional time prediction task to complete effective temporal information within the learned unified representation. Extensive experiments on six benchmark datasets verify the state-of-the-art performance and the effectiveness of our method.