A Cross-Validation Study of Turkish Sentiment Analysis Datasets and Tools

TL;DR

This study reviews Turkish sentiment analysis datasets and tools over a decade, analyzing their characteristics and performance to improve understanding and application in Turkish NLP.

Contribution

It provides a comprehensive taxonomy of Turkish sentiment datasets and evaluates state-of-the-art tools across these datasets, highlighting performance dependencies.

Findings

Performance varies with text characteristics

Datasets exhibit diverse features and quality

Tools show inconsistent results depending on dataset

Abstract

In recent years, sentiment analysis has gained increasing significance, prompting researchers to explore datasets in various languages, including Turkish. However, the limited availability of Turkish datasets has led to their multifaceted usage in different studies, yielding diverse outcomes. To overcome this challenge, a rigorous review was conducted of research articles published between 2012 and 2022. 31 studies were listed, and 23 Turkish datasets obtained from publicly available sources and email requests used in these studies were collected. We labeled these 31 studies using a taxonomy. We provide a map of sentiment analysis datasets according to this taxonomy in Turkish over 10 years. Moreover, we run state-of-the-art sentiment analysis tools on these datasets and analyzed performance across popular Turkish sentiment datasets. We observed that the performance of the sentiment analysis tools significantly depends on the characteristics of the target text. Our study fosters a more nuanced understanding of sentiment analysis in the Turkish language.