How Effectively Do LLMs Extract Feature-Sentiment Pairs from App Reviews?

TL;DR

This paper evaluates the effectiveness of various Large Language Models in extracting feature-specific sentiment pairs from app reviews, comparing their performance across different few-shot learning scenarios.

Contribution

It provides a comprehensive comparison of LLMs like GPT-4, ChatGPT, and Llama-2 variants for feature-sentiment extraction in app reviews, highlighting their strengths and limitations.

Findings

GPT-4 outperforms rule-based methods in feature extraction by 17% in zero-shot.

Fine-tuned RE-BERT surpasses GPT-4 by 6% in feature extraction.

GPT-4 achieves 76% and 45% F1-scores for positive and neutral sentiment prediction in zero-shot.

Abstract

Automatic analysis of user reviews to understand user sentiments toward app functionality (i.e. app features) helps align development efforts with user expectations and needs. Recent advances in Large Language Models (LLMs) such as ChatGPT have shown impressive performance on several new tasks without updating the model's parameters i.e. using zero or a few labeled examples, but the capabilities of LLMs are yet unexplored for feature-specific sentiment analysis. The goal of our study is to explore the capabilities of LLMs to perform feature-specific sentiment analysis of user reviews. This study compares the performance of state-of-the-art LLMs, including GPT-4, ChatGPT, and different variants of Llama-2 chat, against previous approaches for extracting app features and associated sentiments in zero-shot, 1-shot, and 5-shot scenarios. The results indicate that GPT-4 outperforms the rule-based SAFE by 17% in f1-score for extracting app features in the zero-shot scenario, with 5-shot further improving it by 6%. However, the fine-tuned RE-BERT exceeds GPT-4 by 6% in f1-score. For predicting positive and neutral sentiments, GPT-4 achieves f1-scores of 76% and 45% in the zero-shot setting, which improve by 7% and 23% in the 5-shot setting, respectively. Our study conducts a thorough evaluation of both proprietary and open-source LLMs to provide an objective assessment of their performance in extracting feature-sentiment pairs.