Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling

TL;DR

This paper introduces Hi-Guard, a hierarchical, policy-aligned multimodal moderation framework that improves accuracy, interpretability, and policy compliance in content moderation systems.

Contribution

The paper proposes a novel hierarchical moderation pipeline with policy-aligned prompts and a multi-level reward, enhancing transparency and effectiveness over existing methods.

Findings

Achieves superior classification accuracy and generalization.

Enhances interpretability and explanation quality.

Demonstrates effectiveness in real-world deployment.

Abstract

Social platforms have revolutionized information sharing, but also accelerated the dissemination of harmful and policy-violating content. To ensure safety and compliance at scale, moderation systems must go beyond efficiency and offer accuracy and interpretability. However, current approaches largely rely on noisy, label-driven learning, lacking alignment with moderation rules and producing opaque decisions that hinder human review. Therefore, we propose Hierarchical Guard (Hi-Guard), a multimodal moderation framework that introduces a new policy-aligned decision paradigm. The term "Hierarchical" reflects two key aspects of our system design: (1) a hierarchical moderation pipeline, where a lightweight binary model first filters safe content and a stronger model handles fine-grained risk classification; and (2) a hierarchical taxonomy in the second stage, where the model performs path-based classification over a hierarchical taxonomy ranging from coarse to fine-grained levels. To ensure alignment with evolving moderation policies, Hi-Guard directly incorporates rule definitions into the model prompt. To further enhance structured prediction and reasoning, we introduce a multi-level soft-margin reward and optimize with Group Relative Policy Optimization (GRPO), penalizing semantically adjacent misclassifications and improving explanation quality. Extensive experiments and real-world deployment demonstrate that Hi-Guard achieves superior classification accuracy, generalization, and interpretability, paving the way toward scalable, transparent, and trustworthy content safety systems. Code is available at: https://github.com/lianqi1008/Hi-Guard.