Improving Word Representations: A Sub-sampled Unigram Distribution for Negative Sampling

TL;DR

This paper proposes a sub-sampled unigram distribution for negative sampling in Word2Vec, improving word vector quality and understanding by optimizing the noise distribution used during training.

Contribution

It introduces a semantics-based sub-sampling method and a semantics weighted model, enhancing negative sampling and sentence completion performance.

Findings

Improved word vector quality across multiple tasks.

Enhanced negative sampling efficiency and effectiveness.

Significant gains in MSR sentence completion accuracy.

Abstract

Word2Vec is the most popular model for word representation and has been widely investigated in literature. However, its noise distribution for negative sampling is decided by empirical trials and the optimality has always been ignored. We suggest that the distribution is a sub-optimal choice, and propose to use a sub-sampled unigram distribution for better negative sampling. Our contributions include: (1) proposing the concept of semantics quantification and deriving a suitable sub-sampling rate for the proposed distribution adaptive to different training corpora; (2) demonstrating the advantages of our approach in both negative sampling and noise contrastive estimation by extensive evaluation tasks; and (3) proposing a semantics weighted model for the MSR sentence completion task, resulting in considerable improvements. Our work not only improves the quality of word vectors but also benefits current understanding of Word2Vec.