Distilling Fine-grained Sentiment Understanding from Large Language Models

TL;DR

This paper proposes a method to transfer fine-grained sentiment understanding from large language models to smaller models, significantly improving their performance and zero-shot capabilities for sentiment analysis tasks.

Contribution

It introduces a distillation approach from LLMs to SLMs for fine-grained sentiment analysis and develops a benchmark to evaluate both models.

Findings

SLMs achieve a 6% improvement in F1-score after distillation.

Distilled SLMs outperform Llama-2-7b with only 220M parameters.

SLMs gain strong zero-shot sentiment classification abilities.

Abstract

Fine-grained sentiment analysis (FSA) aims to extract and summarize user opinions from vast opinionated text. Recent studies demonstrate that large language models (LLMs) possess exceptional sentiment understanding capabilities. However, directly deploying LLMs for FSA applications incurs high inference costs. Therefore, this paper investigates the distillation of fine-grained sentiment understanding from LLMs into small language models (SLMs). We prompt LLMs to examine and interpret the sentiments of given reviews and then utilize the generated content to pretrain SLMs. Additionally, we develop a comprehensive FSA benchmark to evaluate both SLMs and LLMs. Extensive experiments on this benchmark reveal that: (1) distillation significantly enhances the performance of SLMs in FSA tasks, achieving a 6.00\% improvement in $F_1$-score, and the distilled model can outperform Llama-2-7b with only 220M parameters; (2) distillation equips SLMs with excellent zero-shot sentiment classification capabilities, enabling them to match or even exceed their teacher models. These results suggest that distillation from LLMs is a highly promising direction for FSA. We will release our code, data, and pretrained model weights at https://github.com/HITSZ-HLT/FSA-Distillation.