Breaking Sticks and Ambiguities with Adaptive Skip-gram
Sergey Bartunov, Dmitry Kondrashkin, Anton Osokin, Dmitry Vetrov

TL;DR
The paper introduces the Adaptive Skip-gram model, a nonparametric Bayesian extension that automatically learns multiple word representations to handle ambiguity, improving semantic understanding in word embeddings.
Contribution
It presents a novel nonparametric Bayesian extension of Skip-gram that automatically determines the number of senses per word, addressing limitations of prior multi-prototype models.
Findings
Efficient online variational learning algorithm developed
Demonstrated effectiveness on word-sense induction task
Automatically learns the number of senses for each word
Abstract
Recently proposed Skip-gram model is a powerful method for learning high-dimensional word representations that capture rich semantic relationships between words. However, Skip-gram as well as most prior work on learning word representations does not take into account word ambiguity and maintain only single representation per word. Although a number of Skip-gram modifications were proposed to overcome this limitation and learn multi-prototype word representations, they either require a known number of word meanings or learn them using greedy heuristic approaches. In this paper we propose the Adaptive Skip-gram model which is a nonparametric Bayesian extension of Skip-gram capable to automatically learn the required number of representations for all words at desired semantic resolution. We derive efficient online variational learning algorithm for the model and empirically demonstrate its…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Speech Recognition and Synthesis
