Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Models

TL;DR

This paper introduces CompEffDist, a novel distillation framework for sentiment analysis that improves data efficiency and model performance by automating instruction creation and filtering large-scale data.

Contribution

It proposes attribute-based automatic instruction construction and difficulty-based data filtering to enhance knowledge distillation for lightweight sentiment models.

Findings

3B models match 20x larger teachers' performance

Achieves same accuracy with only 10% of data

Outperforms baseline methods in efficiency

Abstract

Recent efforts leverage knowledge distillation techniques to develop lightweight and practical sentiment analysis models. These methods are grounded in human-written instructions and large-scale user texts. Despite the promising results, two key challenges remain: (1) manually written instructions are limited in diversity and quantity, making them insufficient to ensure comprehensive coverage of distilled knowledge; (2) large-scale user texts incur high computational cost, hindering the practicality of these methods. To this end, we introduce CompEffDist, a comprehensive and efficient distillation framework for sentiment analysis. Our framework consists of two key modules: attribute-based automatic instruction construction and difficulty-based data filtering, which correspondingly tackle the aforementioned challenges. Applying our method across multiple model series (Llama-3, Qwen-3, and Gemma-3), we enable 3B student models to match the performance of 20x larger teacher models on most tasks. In addition, our approach greatly outperforms baseline methods in data efficiency, attaining the same performance level with only 10% of the data.