JAKET: Joint Pre-training of Knowledge Graph and Language Understanding

TL;DR

JAKET is a novel joint pre-training framework that effectively integrates knowledge graphs with language models, enhancing performance on knowledge-aware NLP tasks by mutual assistance of knowledge and language modules.

Contribution

It introduces a new joint pre-training method that models both knowledge graphs and language, enabling better adaptation and understanding in NLP applications.

Findings

Achieves superior performance on knowledge-aware NLP tasks

Enables easy adaptation to unseen knowledge graphs in new domains

Effectively leverages knowledge for improved language understanding

Abstract

Knowledge graphs (KGs) contain rich information about world knowledge, entities and relations. Thus, they can be great supplements to existing pre-trained language models. However, it remains a challenge to efficiently integrate information from KG into language modeling. And the understanding of a knowledge graph requires related context. We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other: the knowledge module produces embeddings for entities in text while the language module generates context-aware initial embeddings for entities and relations in the graph. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains. Experimental results on several knowledge-aware NLP tasks show that our proposed framework achieves superior performance by effectively leveraging knowledge in language understanding.