Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation

TL;DR

This paper investigates how incorporating model uncertainty into content moderation improves collaboration between humans and AI, demonstrating that uncertainty-based strategies outperform traditional methods in efficiency and effectiveness.

Contribution

It introduces new metrics for evaluating moderator-model systems and provides a comprehensive benchmark showing the advantages of uncertainty-aware review strategies.

Findings

Uncertainty-based review strategies outperform toxicity score-based strategies.

Choice of review strategy significantly impacts system performance.

Rigorous metrics are essential for developing effective moderation systems.

Abstract

Content moderation is often performed by a collaboration between humans and machine learning models. However, it is not well understood how to design the collaborative process so as to maximize the combined moderator-model system performance. This work presents a rigorous study of this problem, focusing on an approach that incorporates model uncertainty into the collaborative process. First, we introduce principled metrics to describe the performance of the collaborative system under capacity constraints on the human moderator, quantifying how efficiently the combined system utilizes human decisions. Using these metrics, we conduct a large benchmark study evaluating the performance of state-of-the-art uncertainty models under different collaborative review strategies. We find that an uncertainty-based strategy consistently outperforms the widely used strategy based on toxicity scores, and moreover that the choice of review strategy drastically changes the overall system performance. Our results demonstrate the importance of rigorous metrics for understanding and developing effective moderator-model systems for content moderation, as well as the utility of uncertainty estimation in this domain.