Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies

TL;DR

Moderator is a policy-based system that enables fine-grained moderation of text-to-image models by adjusting model weights based on specified policies, effectively reducing the generation of undesired content.

Contribution

It introduces a novel fine-grained, context-aware model editing approach for content moderation in text-to-image diffusion models, surpassing existing unlearning techniques.

Findings

Moderators can learn policies in about 2.29 iterations.

The system prevents 65% of users from generating moderated content within 15 attempts.

Remaining users require 8.3 times more attempts on average.

Abstract

We present Moderator, a policy-based model management system that allows administrators to specify fine-grained content moderation policies and modify the weights of a text-to-image (TTI) model to make it significantly more challenging for users to produce images that violate the policies. In contrast to existing general-purpose model editing techniques, which unlearn concepts without considering the associated contexts, Moderator allows admins to specify what content should be moderated, under which context, how it should be moderated, and why moderation is necessary. Given a set of policies, Moderator first prompts the original model to generate images that need to be moderated, then uses these self-generated images to reverse fine-tune the model to compute task vectors for moderation and finally negates the original model with the task vectors to decrease its performance in generating moderated content. We evaluated Moderator with 14 participants to play the role of admins and found they could quickly learn and author policies to pass unit tests in approximately 2.29 policy iterations. Our experiment with 32 stable diffusion users suggested that Moderator can prevent 65% of users from generating moderated content under 15 attempts and require the remaining users an average of 8.3 times more attempts to generate undesired content.