TL;DR
This paper reveals that correcting a bug in the CBOW implementation makes it perform as well as Skip-gram in word embedding tasks, challenging previous assumptions about their relative effectiveness.
Contribution
The authors identify and fix a bug in popular CBOW implementations, demonstrating that corrected CBOW matches Skip-gram's performance across multiple tasks.
Findings
Corrected CBOW achieves comparable performance to Skip-gram.
CBOW training is significantly faster than Skip-gram.
Faulty implementations caused underperformance of CBOW in prior studies.
Abstract
Mikolov et al. (2013a) observed that continuous bag-of-words (CBOW) word embeddings tend to underperform Skip-gram (SG) embeddings, and this finding has been reported in subsequent works. We find that these observations are driven not by fundamental differences in their training objectives, but more likely on faulty negative sampling CBOW implementations in popular libraries such as the official implementation, word2vec.c, and Gensim. We show that after correcting a bug in the CBOW gradient update, one can learn CBOW word embeddings that are fully competitive with SG on various intrinsic and extrinsic tasks, while being many times faster to train.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
