ParsBERT: Transformer-based Model for Persian Language Understanding

TL;DR

ParsBERT is a monolingual transformer-based model specifically designed for Persian, achieving state-of-the-art results across multiple NLP tasks by leveraging a large Persian dataset.

Contribution

This paper introduces ParsBERT, a Persian-specific BERT model that outperforms multilingual models and prior approaches in various NLP tasks.

Findings

ParsBERT achieves higher scores than multilingual BERT.

It improves state-of-the-art in Sentiment Analysis, Text Classification, NER.

Utilizes a large, diverse Persian dataset for pre-training.

Abstract

The surge of pre-trained language models has begun a new era in the field of Natural Language Processing (NLP) by allowing us to build powerful language models. Among these models, Transformer-based models such as BERT have become increasingly popular due to their state-of-the-art performance. However, these models are usually focused on English, leaving other languages to multilingual models with limited resources. This paper proposes a monolingual BERT for the Persian language (ParsBERT), which shows its state-of-the-art performance compared to other architectures and multilingual models. Also, since the amount of data available for NLP tasks in Persian is very restricted, a massive dataset for different NLP tasks as well as pre-training the model is composed. ParsBERT obtains higher scores in all datasets, including existing ones as well as composed ones and improves the state-of-the-art performance by outperforming both multilingual BERT and other prior works in Sentiment Analysis, Text Classification and Named Entity Recognition tasks.