Semantics- and Syntax-related Subvectors in the Skip-gram Embeddings

TL;DR

This paper demonstrates that skip-gram word embeddings can be decomposed into semantic and syntactic components, revealing distinct subvector structures that correspond to different linguistic roles.

Contribution

The authors introduce a method to decompose skip-gram embeddings into semantic and syntactic subvectors, providing insight into their linguistic interpretability.

Findings

Embeddings can be separated into semantic and syntactic subvectors.

Semantic and syntactic roles are captured by distinct subvector components.

Decomposition enhances understanding of embedding representations.

Abstract

We show that the skip-gram embedding of any word can be decomposed into two subvectors which roughly correspond to semantic and syntactic roles of the word.