An Empirical Study of Deep Learning Sentiment Detection Tools for Software Engineering in Cross-Platform Settings

TL;DR

This study evaluates deep learning sentiment detection tools in software engineering across different platforms, finding that while DL tools often outperform shallow methods within platforms, rule-based tools can be more effective in cross-platform scenarios, highlighting dataset subjectivity issues.

Contribution

It provides a comprehensive empirical comparison of DL and shallow sentiment detection tools in cross-platform software engineering settings, revealing their relative strengths and limitations.

Findings

BERT4SentiSE generally outperforms other supervised tools in cross-platform settings.

SentistrengthSE often outperforms DL tools in cross-platform scenarios.

Dataset subjectivity impacts the performance consistency of sentiment detection tools.

Abstract

Sentiment detection in software engineering (SE) has shown promise to support diverse development activities. However, given the diversity of SE platforms, SE-specific sentiment detection tools may suffer in performance in cross-platform settings. Recently deep learning (DL)-based SE-specific sentiment detection tools are found to offer superior performance than shallow machine learning (ML) based/rule-based tools. However, it is not known how the DL tools perform in cross-platform settings. In this paper, we study whether SE-specific DL sentiment detectors are more effective than shallow ML-based/rule-based sentiment detectors in cross-platform settings. In three datasets, we study three DL tools (SEntiMoji, BERT4SEntiSE, RNN4SentiSE) and compare those against three baselines: two shallow learning tools (Senti4SD, SentiCR) and one rule-based tool (SentistrengthSE). We find that (1) The deep learning SD tools for SE, BERT4SentiSE outperform other supervised tools in cross-platform settings in most cases, but then the tool is outperformed by the rule-based tool SentistrengthSE in most cases. (2) BERT4SentiSE outperforms SentistrengthSE by large margin in within-platform settings across the three datasets and is only narrowly outperformed by SentiStrengthSE in four out of the six cross-platform settings. This finding offers hope for the feasibility to further improve a pre-trained transformer model like BERT4SentiSE in cross-platform settings. (3) The two best-performing deep learning tools (BERT4SentiSE and SentiMoji) show varying level performance drop across the three datasets. We find that this inconsistency is mainly due to the "subjectivity in annotation" and performance improvement for the studied supervised tools in cross-platform settings may require the fixing of the datasets.