CKG: Dynamic Representation Based on Context and Knowledge Graph

TL;DR

The paper introduces CKG, a dynamic language representation method that integrates context and external knowledge graphs to enhance semantic understanding, achieving state-of-the-art results across multiple NLP tasks.

Contribution

It proposes a novel approach combining corpus-based context with knowledge graph information to improve language representations.

Findings

Achieves SOTA 89.2 on SQuAD

Outperforms SAN, ELMo, and BERT$_{Base}$ on key benchmarks

Effectively leverages external knowledge for better semantics

Abstract

Recently, neural language representation models pre-trained on large corpus can capture rich co-occurrence information and be fine-tuned in downstream tasks to improve the performance. As a result, they have achieved state-of-the-art results in a large range of language tasks. However, there exists other valuable semantic information such as similar, opposite, or other possible meanings in external knowledge graphs (KGs). We argue that entities in KGs could be used to enhance the correct semantic meaning of language sentences. In this paper, we propose a new method CKG: Dynamic Representation Based on \textbf{C}ontext and \textbf{K}nowledge \textbf{G}raph. On the one side, CKG can extract rich semantic information of large corpus. On the other side, it can make full use of inside information such as co-occurrence in large corpus and outside information such as similar entities in KGs. We conduct extensive experiments on a wide range of tasks, including QQP, MRPC, SST-5, SQuAD, CoNLL 2003, and SNLI. The experiment results show that CKG achieves SOTA 89.2 on SQuAD compared with SAN (84.4), ELMo (85.8), and BERT$_{Base}$ (88.5).