A Benchmark Study on Sentiment Analysis for Software Engineering Research

TL;DR

This paper evaluates three sentiment analysis tools tailored for software engineering to understand their performance and reliability, highlighting challenges and insights from misclassified examples.

Contribution

It provides a comprehensive benchmark comparison of domain-specific sentiment analysis tools in software engineering, addressing their effectiveness and limitations.

Findings

Domain-specific tools outperform generic ones

Significant misclassification issues identified

Open challenges in sentiment analysis for software engineering

Abstract

A recent research trend has emerged to identify developers' emotions, by applying sentiment analysis to the content of communication traces left in collaborative development environments. Trying to overcome the limitations posed by using off-the-shelf sentiment analysis tools, researchers recently started to develop their own tools for the software engineering domain. In this paper, we report a benchmark study to assess the performance and reliability of three sentiment analysis tools specifically customized for software engineering. Furthermore, we offer a reflection on the open challenges, as they emerge from a qualitative analysis of misclassified texts.