You Shall Know a Tool by the Traces it Leaves: The Predictability of Sentiment Analysis Tools
Daniel Baumartz, Mevl\"ut Bagci, Alexander Henlein, Maxim Konca, Andy, L\"ucking, Alexander Mehler

TL;DR
This paper demonstrates that sentiment analysis tools produce inconsistent results across datasets and languages, and that their outputs can be used to predict which tool was used, revealing inherent biases.
Contribution
It shows that sentiment analysis tools have predictable biases based on their outputs, challenging their validity as consistent classifiers across datasets and languages.
Findings
Sentiment tools disagree on the same dataset.
Sentiment tool can be predicted from its output with high accuracy.
Highlights the need for systematic evaluation of sentiment analysis tools.
Abstract
If sentiment analysis tools were valid classifiers, one would expect them to provide comparable results for sentiment classification on different kinds of corpora and for different languages. In line with results of previous studies we show that sentiment analysis tools disagree on the same dataset. Going beyond previous studies we show that the sentiment tool used for sentiment annotation can even be predicted from its outcome, revealing an algorithmic bias of sentiment analysis. Based on Twitter, Wikipedia and different news corpora from the English, German and French languages, our classifiers separate sentiment tools with an averaged F1-score of 0.89 (for the English corpora). We therefore warn against taking sentiment annotations as face value and argue for the need of more and systematic NLP evaluation studies.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Text Analysis Techniques · Topic Modeling · Computational and Text Analysis Methods
