MulDE: Multi-teacher Knowledge Distillation for Low-dimensional Knowledge Graph Embeddings

TL;DR

MulDE is a knowledge distillation framework that enhances low-dimensional knowledge graph embeddings by leveraging multiple hyperbolic teacher models, improving performance and training efficiency.

Contribution

Introduces a novel multi-teacher distillation method with iterative strategies and adaptive mechanisms for low-dimensional KGE models.

Findings

Distilled 32-dimensional models outperform some high-dimensional methods.

MulDE improves training speed and accuracy of low-dimensional KGE models.

Effective knowledge transfer from multiple hyperbolic teachers enhances low-dimensional embeddings.

Abstract

Link prediction based on knowledge graph embeddings (KGE) aims to predict new triples to automatically construct knowledge graphs (KGs). However, recent KGE models achieve performance improvements by excessively increasing the embedding dimensions, which may cause enormous training costs and require more storage space. In this paper, instead of training high-dimensional models, we propose MulDE, a novel knowledge distillation framework, which includes multiple low-dimensional hyperbolic KGE models as teachers and two student components, namely Junior and Senior. Under a novel iterative distillation strategy, the Junior component, a low-dimensional KGE model, asks teachers actively based on its preliminary prediction results, and the Senior component integrates teachers' knowledge adaptively to train the Junior component based on two mechanisms: relation-specific scaling and contrast attention. The experimental results show that MulDE can effectively improve the performance and training speed of low-dimensional KGE models. The distilled 32-dimensional model is competitive compared to the state-of-the-art high-dimensional methods on several widely-used datasets.