Word2Bits - Quantized Word Vectors
Maximilian Lam

TL;DR
This paper introduces a method to create highly compressed word vectors using 1-2 bits per parameter, which are memory-efficient, act as regularizers, and outperform full-precision vectors on several NLP tasks.
Contribution
The paper presents a novel quantization approach integrated into Word2Vec, enabling high-quality, extremely compact word vectors that outperform traditional vectors on key NLP benchmarks.
Findings
Quantized vectors use 8-16x less space than full precision vectors.
Quantized vectors outperform full precision vectors on word similarity tasks.
Training with quantization acts as an effective regularizer.
Abstract
Word vectors require significant amounts of memory and storage, posing issues to resource limited devices like mobile phones and GPUs. We show that high quality quantized word vectors using 1-2 bits per parameter can be learned by introducing a quantization function into Word2Vec. We furthermore show that training with the quantization function acts as a regularizer. We train word vectors on English Wikipedia (2017) and evaluate them on standard word similarity and analogy tasks and on question answering (SQuAD). Our quantized word vectors not only take 8-16x less space than full precision (32 bit) word vectors but also outperform them on word similarity tasks and question answering.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
