Leader: Prefixing a Length for Faster Word Vector Serialization
Brian Lester

TL;DR
This paper introduces the Leader format, a new binary file format for word embeddings that combines fast read times with small file sizes, improving upon existing formats like GloVe and word2vec.
Contribution
The paper proposes the Leader format, a novel binary format with length prefixing that enhances speed while maintaining compactness, along with tools for conversion and reading.
Findings
Leader format reduces read times compared to GloVe and word2vec.
Maintains small file size similar to binary formats.
Provides a library for easy format conversion and access.
Abstract
Two competing file formats have become the de facto standards for distributing pre-trained word embeddings. Both are named after the most popular pre-trained embeddings that are distributed in that format. The GloVe format is an entirely text based format that suffers from huge file sizes and slow reads, and the word2vec format is a smaller binary format that mixes a textual representation of words with a binary representation of the vectors themselves. Both formats have problems that we solve with a new format we call the Leader format. We include a word length prefix for faster reads while maintaining the smaller file size a binary format offers. We also created a minimalist library to facilitate the reading and writing of various word vector formats, as well as tools for converting pre-trained embeddings to our new Leader format.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Algorithms and Data Compression
MethodsGloVe Embeddings
