HmBlogs: A big general Persian corpus

TL;DR

This paper introduces hmBlogs, the largest Persian blog corpus with 20 million posts and 6.8 billion tokens, and demonstrates its superiority for language modeling and semantic tasks.

Contribution

The creation of the largest independently prepared Persian corpus and the development of word embedding models and a semantic analogy dataset for Persian.

Findings

hmBlogs outperforms other Persian corpora in language modeling tasks

Preprocessed corpus improves embedding quality and evaluation results

The semantic analogy dataset enables better semantic understanding in Persian

Abstract

This paper introduces the hmBlogs corpus for Persian, as a low resource language. This corpus has been prepared based on a collection of nearly 20 million blog posts over a period of about 15 years from a space of Persian blogs and includes more than 6.8 billion tokens. It can be claimed that this corpus is currently the largest Persian corpus that has been prepared independently for the Persian language. This corpus is presented in both raw and preprocessed forms, and based on the preprocessed corpus some word embedding models are produced. By the provided models, the hmBlogs is compared with some of the most important corpora available in Persian, and the results show the superiority of the hmBlogs corpus over the others. These evaluations also present the importance and effects of corpora, evaluation datasets, model production methods, different hyperparameters and even the evaluation methods. In addition to evaluating the corpus and its produced language models, this research also presents a semantic analogy dataset.