Exploration and Evaluation of Bias in Cyberbullying Detection with Machine Learning

TL;DR

This paper investigates how biases from data collection and labeling affect cyberbullying detection models, emphasizing the importance of dataset curation and cross-dataset evaluation for real-world effectiveness.

Contribution

It provides a detailed analysis of bias sources in cyberbullying datasets and evaluates model generalization across different datasets, highlighting challenges in real-world deployment.

Findings

Models experience a significant drop in Macro F1 Score when tested on unseen datasets.

Biases from data collection and labeling significantly impact model performance.

Cross-dataset evaluation is crucial for assessing real-world applicability.

Abstract

It is well known that the usefulness of a machine learning model is due to its ability to generalize to unseen data. This study uses three popular cyberbullying datasets to explore the effects of data, how it's collected, and how it's labeled, on the resulting machine learning models. The bias introduced from differing definitions of cyberbullying and from data collection is discussed in detail. An emphasis is made on the impact of dataset expansion methods, which utilize current data points to fetch and label new ones. Furthermore, explicit testing is performed to evaluate the ability of a model to generalize to unseen datasets through cross-dataset evaluation. As hypothesized, the models have a significant drop in the Macro F1 Score, with an average drop of 0.222. As such, this study effectively highlights the importance of dataset curation and cross-dataset testing for creating models with real-world applicability. The experiments and other code can be found at https://github.com/rootdrew27/cyberbullying-ml.