Beyond Word Embeddings: Learning Entity and Concept Representations from Large Scale Knowledge Bases

TL;DR

This paper introduces a novel method for learning entity and concept representations by integrating large-scale knowledge bases, achieving state-of-the-art results in analogy reasoning and concept categorization tasks.

Contribution

It proposes a simple, effective technique to combine Wikipedia and Probase knowledge bases into a unified embedding model using skip-gram, improving concept representation quality.

Findings

Achieved 91% accuracy on semantic analogies

Attained 100% and 98% accuracy on concept categorization datasets

Demonstrated effective unsupervised argument type identification

Abstract

Text representations using neural word embeddings have proven effective in many NLP applications. Recent researches adapt the traditional word embedding models to learn vectors of multiword expressions (concepts/entities). However, these methods are limited to textual knowledge bases (e.g., Wikipedia). In this paper, we propose a novel and simple technique for integrating the knowledge about concepts from two large scale knowledge bases of different structure (Wikipedia and Probase) in order to learn concept representations. We adapt the efficient skip-gram model to seamlessly learn from the knowledge in Wikipedia text and Probase concept graph. We evaluate our concept embedding models on two tasks: (1) analogical reasoning, where we achieve a state-of-the-art performance of 91% on semantic analogies, (2) concept categorization, where we achieve a state-of-the-art performance on two benchmark datasets achieving categorization accuracy of 100% on one and 98% on the other. Additionally, we present a case study to evaluate our model on unsupervised argument type identification for neural semantic parsing. We demonstrate the competitive accuracy of our unsupervised method and its ability to better generalize to out of vocabulary entity mentions compared to the tedious and error prone methods which depend on gazetteers and regular expressions.