Exa-PSD: a new Persian sentiment analysis dataset on Twitter

TL;DR

This paper introduces Exa-PSD, a new Persian Twitter sentiment dataset with 12,000 annotated tweets, and evaluates it using pre-trained language models, achieving a Macro F-score of nearly 80%.

Contribution

The paper provides the first large-scale Persian Twitter sentiment dataset with annotations and demonstrates its usefulness with baseline model evaluations.

Findings

Achieved 79.87% Macro F-score using Pars BERT and RoBERTa.

Dataset contains 12,000 tweets annotated in positive, neutral, and negative classes.

Dataset characteristics and statistics are detailed.

Abstract

Today, Social networks such as Twitter are the most widely used platforms for communication of people. Analyzing this data has useful information to recognize the opinion of people in tweets. Sentiment analysis plays a vital role in NLP, which identifies the opinion of the individuals about a specific topic. Natural language processing in Persian has many challenges despite the adventure of strong language models. The datasets available in Persian are generally in special topics such as products, foods, hotels, etc while users may use ironies, colloquial phrases in social media To overcome these challenges, there is a necessity for having a dataset of Persian sentiment analysis on Twitter. In this paper, we introduce the Exa sentiment analysis Persian dataset, which is collected from Persian tweets. This dataset contains 12,000 tweets, annotated by 5 native Persian taggers. The aforementioned data is labeled in 3 classes: positive, neutral and negative. We present the characteristics and statistics of this dataset and use the pre-trained Pars Bert and Roberta as the base model to evaluate this dataset. Our evaluation reached a 79.87 Macro F-score, which shows the model and data can be adequately valuable for a sentiment analysis system.