Fast and Continual Knowledge Graph Embedding via Incremental LoRA

TL;DR

This paper introduces a fast, incremental knowledge graph embedding framework ( extbackslash model) that efficiently learns new knowledge while preserving existing knowledge, significantly reducing training time and maintaining competitive performance.

Contribution

The paper proposes an innovative CKGE framework with an incremental low-rank adapter ( extbackslash mec) that adaptively allocates resources and isolates new knowledge to prevent forgetting.

Findings

Reduces training time by 34-49% on public datasets.

Saves 51-68% training time on new datasets with improved performance.

Achieves competitive link prediction results with state-of-the-art models.

Abstract

Continual Knowledge Graph Embedding (CKGE) aims to efficiently learn new knowledge and simultaneously preserve old knowledge. Dominant approaches primarily focus on alleviating catastrophic forgetting of old knowledge but neglect efficient learning for the emergence of new knowledge. However, in real-world scenarios, knowledge graphs (KGs) are continuously growing, which brings a significant challenge to fine-tuning KGE models efficiently. To address this issue, we propose a fast CKGE framework (\model), incorporating an incremental low-rank adapter (\mec) mechanism to efficiently acquire new knowledge while preserving old knowledge. Specifically, to mitigate catastrophic forgetting, \model\ isolates and allocates new knowledge to specific layers based on the fine-grained influence between old and new KGs. Subsequently, to accelerate fine-tuning, \model\ devises an efficient \mec\ mechanism, which embeds the specific layers into incremental low-rank adapters with fewer training parameters. Moreover, \mec\ introduces adaptive rank allocation, which makes the LoRA aware of the importance of entities and adjusts its rank scale adaptively. We conduct experiments on four public datasets and two new datasets with a larger initial scale. Experimental results demonstrate that \model\ can reduce training time by 34\%-49\% while still achieving competitive link prediction performance against state-of-the-art models on four public datasets (average MRR score of 21.0\% vs. 21.1\%).Meanwhile, on two newly constructed datasets, \model\ saves 51\%-68\% training time and improves link prediction performance by 1.5\%.