When Rules Fall Short: Agent-Driven Discovery of Emerging Content Issues in Short Video Platforms

TL;DR

This paper introduces an automated, multimodal LLM agent-based system for rapidly discovering and addressing emerging content issues on short-video platforms, significantly improving efficiency over manual methods.

Contribution

The paper presents a novel agent-driven approach that automatically identifies new issues, clusters them, and updates annotation policies, enhancing content governance in short-video platforms.

Findings

F1 score for issue discovery improved by over 20%

Reduced problematic video views by approximately 15%

Accelerated annotation policy updates compared to manual discovery

Abstract

Trends on short-video platforms evolve at a rapid pace, with new content issues emerging every day that fall outside the coverage of existing annotation policies. However, traditional human-driven discovery of emerging issues is too slow, which leads to delayed updates of annotation policies and poses a major challenge for effective content governance. In this work, we propose an automatic issue discovery method based on multimodal LLM agents. Our approach automatically recalls short videos containing potential new issues and applies a two-stage clustering strategy to group them, with each cluster corresponding to a newly discovered issue. The agent then generates updated annotation policies from these clusters, thereby extending coverage to these emerging issues. Our agent has been deployed in the real system. Both offline and online experiments demonstrate that this agent-based method significantly improves the effectiveness of emerging-issue discovery (with an F1 score improvement of over 20%) and enhances the performance of subsequent issue governance (reducing the view count of problematic videos by approximately 15%). More importantly, compared to manual issue discovery, it greatly reduces time costs and substantially accelerates the iteration of annotation policies.