A Comprehensive Evaluation of Large Language Models on Aspect-Based Sentiment Analysis

TL;DR

This paper provides a comprehensive evaluation of large language models on aspect-based sentiment analysis, comparing fine-tuning and in-context learning approaches across multiple datasets and subtasks, revealing their competitive performance.

Contribution

It introduces a unified task formulation and evaluation framework for LLMs in ABSA, including novel demonstration selection strategies and multi-task instruction fine-tuning, advancing the understanding of LLM capabilities in ABSA.

Findings

LLMs outperform fine-tuned small models in fine-tuning-dependent ABSA tasks.

In-context learning enables LLMs to compete with fine-tuned models on some ABSA subtasks.

Proposed strategies improve LLM performance in few-shot settings.

Abstract

Recently, Large Language Models (LLMs) have garnered increasing attention in the field of natural language processing, revolutionizing numerous downstream tasks with powerful reasoning and generation abilities. For example, In-Context Learning (ICL) introduces a fine-tuning-free paradigm, allowing out-of-the-box LLMs to execute downstream tasks by analogy learning without any fine-tuning. Besides, in a fine-tuning-dependent paradigm where substantial training data exists, Parameter-Efficient Fine-Tuning (PEFT), as the cost-effective methods, enable LLMs to achieve excellent performance comparable to full fine-tuning. However, these fascinating techniques employed by LLMs have not been fully exploited in the ABSA field. Previous works probe LLMs in ABSA by merely using randomly selected input-output pairs as demonstrations in ICL, resulting in an incomplete and superficial evaluation. In this paper, we shed light on a comprehensive evaluation of LLMs in the ABSA field, involving 13 datasets, 8 ABSA subtasks, and 6 LLMs. Specifically, we design a unified task formulation to unify ``multiple LLMs for multiple ABSA subtasks in multiple paradigms.'' For the fine-tuning-dependent paradigm, we efficiently fine-tune LLMs using instruction-based multi-task learning. For the fine-tuning-free paradigm, we propose 3 demonstration selection strategies to stimulate the few-shot abilities of LLMs. Our extensive experiments demonstrate that LLMs achieve a new state-of-the-art performance compared to fine-tuned Small Language Models (SLMs) in the fine-tuning-dependent paradigm. More importantly, in the fine-tuning-free paradigm where SLMs are ineffective, LLMs with ICL still showcase impressive potential and even compete with fine-tuned SLMs on some ABSA subtasks.