Psychological State in Text: A Limitation of Sentiment Analysis

TL;DR

This study investigates whether sentiment analysis models can accurately predict psychological states from text, revealing that while models predict scores well, these scores do not correlate with human self-assessments of sentiment.

Contribution

The paper highlights the limitations of current sentiment analysis models in capturing true psychological states, emphasizing the gap between model predictions and human emotional self-awareness.

Findings

Sentiment scores do not correlate with human self-checked sentiment.

Models perform well at predicting scores but lack psychological validity.

Psychological states are not reliably inferred from sentiment analysis alone.

Abstract

Starting with the idea that sentiment analysis models should be able to predict not only positive or negative but also other psychological states of a person, we implement a sentiment analysis model to investigate the relationship between the model and emotional state. We first examine psychological measurements of 64 participants and ask them to write a book report about a story. After that, we train our sentiment analysis model using crawled movie review data. We finally evaluate participants' writings, using the pretrained model as a concept of transfer learning. The result shows that sentiment analysis model performs good at predicting a score, but the score does not have any correlation with human's self-checked sentiment.