Incivility Detection in Open Source Code Review and Issue Discussions

TL;DR

This study compares classical machine learning models and BERT for detecting incivility in open source discussions, finding BERT outperforms others but adding context does not improve performance, with challenges in cross-platform accuracy.

Contribution

The paper evaluates the effectiveness of BERT versus classical models for incivility detection and explores the impact of contextual information and cross-platform performance.

Findings

BERT achieves the highest F1-score of 0.95.

Classical models underperform in detecting civil discussions.

Adding context does not improve BERT's performance.

Abstract

Given the democratic nature of open source development, code review and issue discussions may be uncivil. Incivility, defined as features of discussion that convey an unnecessarily disrespectful tone, can have negative consequences to open source communities. To prevent or minimize these negative consequences, open source platforms have included mechanisms for removing uncivil language from the discussions. However, such approaches require manual inspection, which can be overwhelming given the large number of discussions. To help open source communities deal with this problem, in this paper, we aim to compare six classical machine learning models with BERT to detect incivility in open source code review and issue discussions. Furthermore, we assess if adding contextual information improves the models' performance and how well the models perform in a cross-platform setting. We found that BERT performs better than classical machine learning models, with a best F1-score of 0.95. Furthermore, classical machine learning models tend to underperform to detect non-technical and civil discussions. Our results show that adding the contextual information to BERT did not improve its performance and that none of the analyzed classifiers had an outstanding performance in a cross-platform setting. Finally, we provide insights into the tones that the classifiers misclassify.