SubGram: Extending Skip-gram Word Representation with Substrings

TL;DR

SubGram enhances the Skip-gram word embedding model by incorporating word substrings, leading to improved performance in capturing linguistic information without additional supervision.

Contribution

It introduces a novel extension to Skip-gram that considers word structure, significantly improving embedding quality.

Findings

Achieves large gains over original Skip-gram on test set

Effectively captures syntactic and semantic information

Demonstrates the benefit of modeling word structure

Abstract

Skip-gram (word2vec) is a recent method for creating vector representations of words ("distributed word representations") using a neural network. The representation gained popularity in various areas of natural language processing, because it seems to capture syntactic and semantic information about words without any explicit supervision in this respect. We propose SubGram, a refinement of the Skip-gram model to consider also the word structure during the training process, achieving large gains on the Skip-gram original test set.