GMP: A Benchmark for Content Moderation under Co-occurring Violations and Dynamic Rules

TL;DR

This paper introduces GMP, a benchmark designed to evaluate AI content moderation systems' ability to handle multiple simultaneous violations and adapt to changing moderation rules, addressing limitations of current models in real-world scenarios.

Contribution

The paper presents GMP, a new benchmark that tests AI moderation under co-occurring violations and dynamic rules, highlighting gaps in existing evaluation methods.

Findings

Current LLMs struggle with co-occurring violations.

Dynamic rules significantly impact moderation accuracy.

GMP reveals limitations of existing models in complex scenarios.

Abstract

Online content moderation is essential for maintaining a healthy digital environment, and reliance on AI for this task continues to grow. Consider a user comment using national stereotypes to insult a politician. This example illustrates two critical challenges in real-world scenarios: (1) Co-occurring Violations, where a single post violates multiple policies (e.g., prejudice and personal attacks); (2) Dynamic rules of moderation, where determination of a violation depends on platform-specific guidelines that evolve across contexts . The intersection of co-occurring harms and dynamically changing rules highlights a core limitation of current AI systems: although large language models (LLMs) are adept at following fixed guidelines, their judgment capabilities degrade when policies are unstable or context-dependent . In practice, such shortcomings lead to inconsistent moderation: either erroneously restricting legitimate expression or allowing harmful content to remain online . This raises a critical question for evaluation: Does high performance on existing static benchmarks truly guarantee robust generalization of AI judgment to real-world scenarios involving co-occurring violations and dynamically changing rules?