Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go?

TL;DR

This paper evaluates the potential of zero-shot learning techniques for sentiment analysis in software engineering, showing they can perform comparably to fine-tuned models and address data annotation challenges.

Contribution

It provides an empirical assessment of various ZSL methods in software engineering sentiment analysis, highlighting their effectiveness and limitations.

Findings

ZSL techniques with expert labels achieve macro-F1 scores similar to fine-tuned transformers.

Subjectivity and polar facts are main causes of misclassification in ZSL.

ZSL reduces reliance on annotated datasets for sentiment analysis in software engineering.

Abstract

Sentiment analysis in software engineering focuses on understanding emotions expressed in software artifacts. Previous research highlighted the limitations of applying general off-the-shelf sentiment analysis tools within the software engineering domain and indicated the need for specialized tools tailored to various software engineering contexts. The development of such tools heavily relies on supervised machine learning techniques that necessitate annotated datasets. Acquiring such datasets is a substantial challenge, as it requires domain-specific expertise and significant effort. Objective: This study explores the potential of ZSL to address the scarcity of annotated datasets in sentiment analysis within software engineering Method:} We conducted an empirical experiment to evaluate the performance of various ZSL techniques, including embedding-based, NLI-based, TARS-based, and generative-based ZSL techniques. We assessed the performance of these techniques under different labels setups to examine the impact of label configurations. Additionally, we compared the results of the ZSL techniques with state-of-the-art fine-tuned transformer-based models. Finally, we performed an error analysis to identify the primary causes of misclassifications. Results: Our findings demonstrate that ZSL techniques, particularly those combining expert-curated labels with embedding-based or generative-based models, can achieve macro-F1 scores comparable to fine-tuned transformer-based models. The error analysis revealed that subjectivity in annotation and polar facts are the main contributors to ZSL misclassifications. Conclusion: This study demonstrates the potential of ZSL for sentiment analysis in software engineering. ZSL can provide a solution to the challenge of annotated dataset scarcity by reducing reliance on annotated dataset.