PRJ: Perception-Retrieval-Judgement for Generated Images

TL;DR

PRJ is a structured, cognitively inspired framework for detecting nuanced and implicit harms in AI-generated images, improving interpretability and accuracy over existing safety systems.

Contribution

It introduces a three-stage perception-retrieval-judgement process that models toxicity detection as structured reasoning, enhancing interpretability and handling nuanced harms.

Findings

PRJ outperforms existing safety checkers in detection accuracy.

PRJ provides improved robustness against adversarial manipulations.

PRJ enables categorical and interpretative analysis of harms.

Abstract

The rapid progress of generative AI has enabled remarkable creative capabilities, yet it also raises urgent concerns regarding the safety of AI-generated visual content in real-world applications such as content moderation, platform governance, and digital media regulation. This includes unsafe material such as sexually explicit images, violent scenes, hate symbols, propaganda, and unauthorized imitations of copyrighted artworks. Existing image safety systems often rely on rigid category filters and produce binary outputs, lacking the capacity to interpret context or reason about nuanced, adversarially induced forms of harm. In addition, standard evaluation metrics (e.g., attack success rate) fail to capture the semantic severity and dynamic progression of toxicity. To address these limitations, we propose Perception-Retrieval-Judgement (PRJ), a cognitively inspired framework that models toxicity detection as a structured reasoning process. PRJ follows a three-stage design: it first transforms an image into descriptive language (perception), then retrieves external knowledge related to harm categories and traits (retrieval), and finally evaluates toxicity based on legal or normative rules (judgement). This language-centric structure enables the system to detect both explicit and implicit harms with improved interpretability and categorical granularity. In addition, we introduce a dynamic scoring mechanism based on a contextual toxicity risk matrix to quantify harmfulness across different semantic dimensions. Experiments show that PRJ surpasses existing safety checkers in detection accuracy and robustness while uniquely supporting structured category-level toxicity interpretation.