Improving Continual Learning of Knowledge Graph Embeddings via Informed Initialization

TL;DR

This paper introduces an informed initialization method for knowledge graph embeddings in continual learning, improving accuracy, retention, and training efficiency during frequent updates.

Contribution

It proposes a novel initialization strategy that leverages KG schema and class information to enhance continual learning of KGEs, reducing catastrophic forgetting and training time.

Findings

Improves predictive performance of KGEs in continual learning.

Enhances knowledge retention during updates.

Reduces training epochs and time for incremental learning.

Abstract

Many Knowledege Graphs (KGs) are frequently updated, forcing their Knowledge Graph Embeddings (KGEs) to adapt to these changes. To address this problem, continual learning techniques for KGEs incorporate embeddings for new entities while updating the old ones. One necessary step in these methods is the initialization of the embeddings, as an input to the KGE learning process, which can have an important impact in the accuracy of the final embeddings, as well as in the time required to train them. This is especially relevant for relatively small and frequent updates. We propose a novel informed embedding initialization strategy, which can be seamlessly integrated into existing continual learning methods for KGE, that enhances the acquisition of new knowledge while reducing catastrophic forgetting. Specifically, the KG schema and the previously learned embeddings are utilized to obtain initial representations for the new entities, based on the classes the entities belong to. Our extensive experimental analysis shows that the proposed initialization strategy improves the predictive performance of the resulting KGEs, while also enhancing knowledge retention. Furthermore, our approach accelerates knowledge acquisition, reducing the number of epochs, and therefore time, required to incrementally learn new embeddings. Finally, its benefits across various types of KGE learning models are demonstrated.