Assessment of Massively Multilingual Sentiment Classifiers

TL;DR

This paper evaluates 11 multilingual sentiment classifiers across 80 datasets in 27 languages, emphasizing the importance of balanced multilingual performance over marginal SOTA improvements, and offers best practices for handling large multilingual datasets.

Contribution

It introduces the largest unified multilingual sentiment dataset collection and provides a comprehensive evaluation of models, highlighting the significance of balanced multilingual performance.

Findings

Smaller, faster models can perform comparably to larger ones in multilingual sentiment tasks.

Balanced performance across languages is more crucial than SOTA results on individual languages.

Best practices for managing large multilingual sentiment datasets are proposed.

Abstract

Models are increasing in size and complexity in the hunt for SOTA. But what if those 2\% increase in performance does not make a difference in a production use case? Maybe benefits from a smaller, faster model outweigh those slight performance gains. Also, equally good performance across languages in multilingual tasks is more important than SOTA results on a single one. We present the biggest, unified, multilingual collection of sentiment analysis datasets. We use these to assess 11 models and 80 high-quality sentiment datasets (out of 342 raw datasets collected) in 27 languages and included results on the internally annotated datasets. We deeply evaluate multiple setups, including fine-tuning transformer-based models for measuring performance. We compare results in numerous dimensions addressing the imbalance in both languages coverage and dataset sizes. Finally, we present some best practices for working with such a massive collection of datasets and models from a multilingual perspective.