Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models

TL;DR

This paper demonstrates that Chain-of-Thought prompting significantly improves the accuracy of granular sentiment classification in app reviews by enabling large language models to reason explicitly about nuanced user feedback.

Contribution

It introduces the application of Chain-of-Thought prompting to enhance granular sentiment analysis in large language models, showing substantial accuracy improvements over simple prompting.

Findings

CoT prompting increased accuracy from 84% to 93%.

Explicit reasoning benefits sentiment analysis performance.

Evaluation on 2000 Amazon app reviews.

Abstract

We explore the use of Chain-of-Thought (CoT) prompting with large language models (LLMs) to improve the accuracy of granular sentiment categorization in app store reviews. Traditional numeric and polarity-based ratings often fail to capture the nuanced sentiment embedded in user feedback. We evaluated the effectiveness of CoT prompting versus simple prompting on 2000 Amazon app reviews by comparing each method's predictions to human judgements. CoT prompting improved classification accuracy from 84% to 93% highlighting the benefit of explicit reasoning in enhancing sentiment analysis performance.