# Sentiment analysis is not solved! Assessing and probing sentiment classification

## TL;DR

This paper investigates the remaining challenges in sentiment analysis by creating a dataset of difficult sentences annotated for linguistic phenomena, enabling better understanding of classifier limitations.

## Contribution

It introduces a challenging dataset of misclassified sentences annotated for linguistic phenomena and demonstrates its use for probing sentiment classifier performance.

## Key findings

- Identified key linguistic phenomena affecting sentiment classification.
- Provided a dataset for analyzing classifier errors.
- Showed how the dataset can be used to improve sentiment models.

## Abstract

Neural methods for SA have led to quantitative improvements over previous approaches, but these advances are not always accompanied with a thorough analysis of the qualitative differences. Therefore, it is not clear what outstanding conceptual challenges for sentiment analysis remain. In this work, we attempt to discover what challenges still prove a problem for sentiment classifiers for English and to provide a challenging dataset. We collect the subset of sentences that an (oracle) ensemble of state-of-the-art sentiment classifiers misclassify and then annotate them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, modality, etc. The dataset is available at https://github.com/ltgoslo/assessing_and_probing_sentiment. Finally, we provide a case study that demonstrates the usefulness of the dataset to probe the performance of a given sentiment classifier with respect to linguistic phenomena.