Bandits for Online Calibration: An Application to Content Moderation on Social Media Platforms

TL;DR

This paper presents a bandit-based system for calibrating risk models in Meta's content moderation, dynamically adapting to changing violation trends to improve moderation effectiveness.

Contribution

It introduces a contextual bandit approach to calibrate multiple risk models for content moderation, addressing temporal changes and model reliability in a production environment.

Findings

Increased moderation effectiveness by 13%

Successfully handles changing and new risk models

Demonstrates real-world applicability in Meta's platform

Abstract

We describe the current content moderation strategy employed by Meta to remove policy-violating content from its platforms. Meta relies on both handcrafted and learned risk models to flag potentially violating content for human review. Our approach aggregates these risk models into a single ranking score, calibrating them to prioritize more reliable risk models. A key challenge is that violation trends change over time, affecting which risk models are most reliable. Our system additionally handles production challenges such as changing risk models and novel risk models. We use a contextual bandit to update the calibration in response to such trends. Our approach increases Meta's top-line metric for measuring the effectiveness of its content moderation strategy by 13%.