The Limits of ChatGPT in Extracting Aspect-Category-Opinion-Sentiment Quadruples: A Comparative Analysis

TL;DR

This paper evaluates ChatGPT's ability to extract aspect-category-opinion-sentiment quadruples from texts, developing specialized prompts and few-shot learning strategies, and compares its performance with state-of-the-art models across multiple datasets.

Contribution

It introduces a specialized prompt template and a selection method for few-shot examples to enhance ChatGPT's performance on complex quadruple extraction tasks.

Findings

ChatGPT's performance is limited compared to specialized models.

Prompt engineering and few-shot examples improve extraction effectiveness.

ChatGPT's capabilities have boundaries in complex sentiment analysis tasks.

Abstract

Recently, ChatGPT has attracted great attention from both industry and academia due to its surprising abilities in natural language understanding and generation. We are particularly curious about whether it can achieve promising performance on one of the most complex tasks in aspect-based sentiment analysis, i.e., extracting aspect-category-opinion-sentiment quadruples from texts. To this end, in this paper we develop a specialized prompt template that enables ChatGPT to effectively tackle this complex quadruple extraction task. Further, we propose a selection method on few-shot examples to fully exploit the in-context learning ability of ChatGPT and uplift its effectiveness on this complex task. Finally, we provide a comparative evaluation on ChatGPT against existing state-of-the-art quadruple extraction models based on four public datasets and highlight some important findings regarding the capability boundaries of ChatGPT in the quadruple extraction.