Matina: A Large-Scale 73B Token Persian Text Corpus

TL;DR

Matina is a large-scale, high-quality Persian text corpus of 73 billion tokens designed to support the development of NLP models and open-source LLMs for Persian, addressing previous resource limitations.

Contribution

This paper introduces the Matina corpus, a comprehensive 73-billion-token Persian dataset with preprocessing and deduplication, enhancing resources for Persian NLP research.

Findings

The dataset improves model training quality for Persian NLP tasks.

Transformer models trained on Matina achieve competitive performance.

Public availability fosters further research and development in Persian NLP.

Abstract

Text corpora are essential for training models used in tasks like summarization, translation, and large language models (LLMs). While various efforts have been made to collect monolingual and multilingual datasets in many languages, Persian has often been underrepresented due to limited resources for data collection and preprocessing. Existing Persian datasets are typically small and lack content diversity, consisting mainly of weblogs and news articles. This shortage of high-quality, varied data has slowed the development of NLP models and open-source LLMs for Persian. Since model performance depends heavily on the quality of training data, we address this gap by introducing the Matina corpus, a new Persian dataset of 72.9B tokens, carefully preprocessed and deduplicated to ensure high data quality. We further assess its effectiveness by training and evaluating transformer-based models on key NLP tasks. Both the dataset and preprocessing codes are publicly available, enabling researchers to build on and improve this resource for future Persian NLP advancements.