Learning Word Vectors for 157 Languages

TL;DR

This paper presents the training of high-quality word vectors for 157 languages using Wikipedia and Common Crawl data, introducing new evaluation datasets and demonstrating strong performance across multiple languages.

Contribution

It is the first large-scale effort to produce and evaluate word embeddings for 157 languages, including new analogy datasets for French, Hindi, and Polish.

Findings

Strong performance on existing evaluation datasets

Introduction of new analogy datasets for three languages

Coverage of 157 languages with high-quality embeddings

Abstract

Distributed word representations, or word vectors, have recently been applied to many tasks in natural language processing, leading to state-of-the-art performance. A key ingredient to the successful application of these representations is to train them on very large corpora, and use these pre-trained models in downstream tasks. In this paper, we describe how we trained such high quality word representations for 157 languages. We used two sources of data to train these models: the free online encyclopedia Wikipedia and data from the common crawl project. We also introduce three new word analogy datasets to evaluate these word vectors, for French, Hindi and Polish. Finally, we evaluate our pre-trained word vectors on 10 languages for which evaluation datasets exists, showing very strong performance compared to previous models.