Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks

TL;DR

This paper introduces SynGCN and SemGCN, two graph convolutional network-based methods that incorporate syntactic and semantic information into word embeddings, improving their quality without increasing vocabulary size.

Contribution

The paper presents SynGCN and SemGCN, novel GCN-based models that effectively integrate syntactic and semantic context into word embeddings, overcoming vocabulary explosion issues.

Findings

SynGCN outperforms existing embedding methods on various tasks.

SynGCN provides an advantage when combined with ELMo.

SemGCN enhances embeddings with diverse semantic knowledge.

Abstract

Word embeddings have been widely adopted across several NLP applications. Most existing word embedding methods utilize sequential context of a word to learn its embedding. While there have been some attempts at utilizing syntactic context of a word, such methods result in an explosion of the vocabulary size. In this paper, we overcome this problem by proposing SynGCN, a flexible Graph Convolution based method for learning word embeddings. SynGCN utilizes the dependency context of a word without increasing the vocabulary size. Word embeddings learned by SynGCN outperform existing methods on various intrinsic and extrinsic tasks and provide an advantage when used with ELMo. We also propose SemGCN, an effective framework for incorporating diverse semantic knowledge for further enhancing learned word representations. We make the source code of both models available to encourage reproducible research.