Compressing Word Embeddings

TL;DR

This paper explores methods to compress word embeddings, significantly reducing memory usage while maintaining performance, and introduces binary factorization for improved interpretability of the compressed representations.

Contribution

It presents a simple compression scheme that reduces memory by a factor of 10 with minimal performance loss and investigates binary factorization for more interpretable embeddings.

Findings

Memory footprint reduced by a factor of 10

Minimal impact on semantic and syntactic performance

Binary factorization enhances interpretability

Abstract

Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic. However, these vector space representations (created through large-scale text analysis) are typically stored verbatim, since their internal structure is opaque. Using word-analogy tests to monitor the level of detail stored in compressed re-representations of the same vector space, the trade-offs between the reduction in memory usage and expressiveness are investigated. A simple scheme is outlined that can reduce the memory footprint of a state-of-the-art embedding by a factor of 10, with only minimal impact on performance. Then, using the same `bit budget', a binary (approximate) factorisation of the same space is also explored, with the aim of creating an equivalent representation with better interpretability.