How do you feel? Measuring User-Perceived Value for Rejecting Machine Decisions in Hate Speech Detection

TL;DR

This paper introduces a value-sensitive rejection mechanism for hate speech detection systems that considers user perceptions to improve human-AI collaboration, using Magnitude Estimation to measure user agreement and guide decision-making.

Contribution

It proposes a novel rejection mechanism based on user-perceived value and introduces Magnitude Estimation as a measurement tool for user agreement in hate speech detection.

Findings

Magnitude Estimation reliably measures user perception.

User-perceived value guides optimal rejection of machine decisions.

Model selection based on user perception outperforms accuracy-based selection.

Abstract

Hate speech moderation remains a challenging task for social media platforms. Human-AI collaborative systems offer the potential to combine the strengths of humans' reliability and the scalability of machine learning to tackle this issue effectively. While methods for task handover in human-AI collaboration exist that consider the costs of incorrect predictions, insufficient attention has been paid to accurately estimating these costs. In this work, we propose a value-sensitive rejection mechanism that automatically rejects machine decisions for human moderation based on users' value perceptions regarding machine decisions. We conduct a crowdsourced survey study with 160 participants to evaluate their perception of correct and incorrect machine decisions in the domain of hate speech detection, as well as occurrences where the system rejects making a prediction. Here, we introduce Magnitude Estimation, an unbounded scale, as the preferred method for measuring user (dis)agreement with machine decisions. Our results show that Magnitude Estimation can provide a reliable measurement of participants' perception of machine decisions. By integrating user-perceived value into human-AI collaboration, we further show that it can guide us in 1) determining when to accept or reject machine decisions to obtain the optimal total value a model can deliver and 2) selecting better classification models as compared to the more widely used target of model accuracy.