Are You Sure You're Positive? Consolidating Chain-of-Thought Agents with Uncertainty Quantification for Aspect-Category Sentiment Analysis

TL;DR

This paper introduces a method that combines multiple chain-of-thought agents with uncertainty quantification in large language models to improve aspect-category sentiment analysis, especially in low-data, domain-transfer scenarios.

Contribution

It proposes novel techniques leveraging token-level uncertainty scores to consolidate chain-of-thought agents, enhancing zero-shot sentiment analysis across domains.

Findings

Effective in low-data and domain transfer settings

Demonstrates utility of token-level uncertainty for model aggregation

Shows promising results with Llama and Qwen models

Abstract

Aspect-category sentiment analysis provides granular insights by identifying specific themes within product reviews that are associated with particular opinions. Supervised learning approaches dominate the field. However, data is scarce and expensive to annotate for new domains. We argue that leveraging large language models in a zero-shot setting is beneficial where the time and resources required for dataset annotation are limited. Furthermore, annotation bias may lead to strong results using supervised methods but transfer poorly to new domains in contexts that lack annotations and demand reproducibility. In our work, we propose novel techniques that combine multiple chain-of-thought agents by leveraging large language models' token-level uncertainty scores. We experiment with the 3B and 70B+ parameter size variants of Llama and Qwen models, demonstrating how these approaches can fulfil practical needs and opening a discussion on how to gauge accuracy in label-scarce conditions.