Analyzing Toxicity in Open Source Software Communications Using Psycholinguistics and Moral Foundations Theory

TL;DR

This paper explores a machine learning approach that uses psycholinguistic and moral foundation features to improve toxicity detection in open source software communications, addressing the unique language styles of these channels.

Contribution

It introduces a novel toxicity detection method leveraging moral and psycholinguistic features, achieving up to 7% better F1 scores over existing detectors in OSS contexts.

Findings

Moral features outperform linguistic cues in toxicity detection.

Achieved up to 67.50% F1 in code reviews and 64.83% in issue comments.

Context-specific models are crucial for effective toxicity detection in OSS.

Abstract

Studies have shown that toxic behavior can cause contributors to leave, and hinder newcomers' (especially from underrepresented communities) participation in Open Source Software (OSS) projects. Thus, detection of toxic language plays a crucial role in OSS collaboration and inclusivity. Off-the-shelf toxicity detectors are ineffective when applied to OSS communications, due to the distinct nature of toxicity observed in these channels (e.g., entitlement and arrogance are more frequently observed on GitHub than on Reddit or Twitter). In this paper, we investigate a machine learning-based approach for the automatic detection of toxic communications in OSS. We leverage psycholinguistic lexicons, and Moral Foundations Theory to analyze toxicity in two types of OSS communication channels; issue comments and code reviews. Our evaluation indicates that our approach can achieve a significant performance improvement (up to 7% increase in F1 score) over the existing domain-specific toxicity detector. We found that using moral values as features is more effective than linguistic cues, resulting in 67.50% F1-measure in identifying toxic instances in code review data and 64.83% in issue comments. While the detection accuracy is far from accurate, this improvement demonstrates the potential of integrating moral and psycholinguistic features in toxicity detection models. These findings highlight the importance of context-specific models that consider the unique communication styles within OSS, where interpersonal and value-driven language dynamics differ markedly from general social media platforms. Future work could focus on refining these models to further enhance detection accuracy, possibly by incorporating community-specific norms and conversational context to better capture the nuanced expressions of toxicity in OSS environments.