What is Sentiment Meant to Mean to Language Models?

TL;DR

This paper examines how large language models interpret and classify sentiment, revealing that clearer, more specific prompts lead to more accurate and meaningful results, and advocates for precise measurement in sentiment analysis.

Contribution

It highlights the ambiguity of sentiment as a concept and demonstrates that specifying dimensions improves classification accuracy in language models.

Findings

Sentiment labels correlate strongly with valence.

Classification accuracy improves with precise prompts.

Encourages moving beyond vague sentiment labels for better analysis.

Abstract

Sentiment analysis is one of the most widely used techniques in text analysis. Recent advancements with Large Language Models have made it more accurate and accessible than ever, allowing researchers to classify text with only a plain English prompt. However, "sentiment" entails a wide variety of concepts depending on the domain and tools used. It has been used to mean emotion, opinions, market movements, or simply a general ``good-bad'' dimension. This raises a question: What exactly are language models doing when prompted to label documents by sentiment? This paper first overviews how sentiment is defined across different contexts, highlighting that it is a confounded measurement construct in that it entails multiple variables, such as emotional valence and opinion, without disentangling them. I then test three language models across two data sets with prompts requesting sentiment, valence, and stance classification. I find that sentiment labels most strongly correlate with valence labels. I further find that classification improves when researchers more precisely specify their dimension of interest rather than using the less well-defined concept of sentiment. I conclude by encouraging researchers to move beyond "sentiment" when feasible and use a more precise measurement construct.