Integrate Temporal Graph Learning into LLM-based Temporal Knowledge Graph Model

TL;DR

This paper introduces TGL-LLM, a novel framework that integrates temporal graph learning into large language models to improve reasoning and forecasting in temporal knowledge graphs, addressing previous limitations.

Contribution

The paper proposes a new method combining temporal graph learning with LLMs, including hybrid tokenization and a two-stage training process, to better model temporal and relational patterns in TKGs.

Findings

Outperforms state-of-the-art methods on three real-world datasets.

Effectively captures temporal and relational patterns in TKGs.

Improves reasoning accuracy in temporal knowledge graph forecasting.

Abstract

Temporal Knowledge Graph Forecasting (TKGF) aims to predict future events based on the observed events in history. Recently, Large Language Models (LLMs) have exhibited remarkable capabilities, generating significant research interest in their application for reasoning over temporal knowledge graphs (TKGs). Existing LLM-based methods have integrated retrieved historical facts or static graph representations into LLMs. Despite the notable performance of LLM-based methods, they are limited by the insufficient modeling of temporal patterns and ineffective cross-modal alignment between graph and language, hindering the ability of LLMs to fully grasp the temporal and structural information in TKGs. To tackle these issues, we propose a novel framework TGL-LLM to integrate temporal graph learning into LLM-based temporal knowledge graph model. Specifically, we introduce temporal graph learning to capture the temporal and relational patterns and obtain the historical graph embedding. Furthermore, we design a hybrid graph tokenization to sufficiently model the temporal patterns within LLMs. To achieve better alignment between graph and language, we employ a two-stage training paradigm to finetune LLMs on high-quality and diverse data, thereby resulting in better performance. Extensive experiments on three real-world datasets show that our approach outperforms a range of state-of-the-art (SOTA) methods.