Reassessing the Role of Chain-of-Thought in Sentiment Analysis: Insights and Limitations

TL;DR

This paper investigates whether chain-of-thought reasoning techniques improve sentiment analysis in large language models, finding minimal impact and emphasizing the importance of demonstrations over reasoning processes.

Contribution

It provides an empirical assessment of chain-of-thought prompting in sentiment analysis, highlighting its limited effectiveness and the reliance on demonstrations.

Findings

Chain-of-thought has minimal impact on sentiment analysis.

Sentiment understanding depends mainly on demonstrations.

Models focus on aspect terms rather than sentiment in generated content.

Abstract

The relationship between language and thought remains an unresolved philosophical issue. Existing viewpoints can be broadly categorized into two schools: one asserting their independence, and another arguing that language constrains thought. In the context of large language models, this debate raises a crucial question: Does a language model's grasp of semantic meaning depend on thought processes? To explore this issue, we investigate whether reasoning techniques can facilitate semantic understanding. Specifically, we conceptualize thought as reasoning, employ chain-of-thought prompting as a reasoning technique, and examine its impact on sentiment analysis tasks. The experiments show that chain-of-thought has a minimal impact on sentiment analysis tasks. Both the standard and chain-of-thought prompts focus on aspect terms rather than sentiment in the generated content. Furthermore, counterfactual experiments reveal that the model's handling of sentiment tasks primarily depends on information from demonstrations. The experimental results support the first viewpoint.