Moderating Illicit Online Image Promotion for Unsafe User-Generated Content Games Using Large Vision-Language Models

TL;DR

This paper introduces UGCG-Guard, a novel system using large vision-language models and zero-shot domain adaptation to effectively detect illicit promotional images for unsafe user-generated content games, achieving 94% accuracy.

Contribution

The study presents the first dataset of illicit UGCG promotion images and develops UGCG-Guard, a system that leverages large vision-language models with a novel prompting strategy for zero-shot detection.

Findings

UGCG-Guard achieves 94% detection accuracy.

A new dataset of 2,924 illicit promotional images is introduced.

The approach demonstrates effective zero-shot domain adaptation.

Abstract

Online user generated content games (UGCGs) are increasingly popular among children and adolescents for social interaction and more creative online entertainment. However, they pose a heightened risk of exposure to explicit content, raising growing concerns for the online safety of children and adolescents. Despite these concerns, few studies have addressed the issue of illicit image-based promotions of unsafe UGCGs on social media, which can inadvertently attract young users. This challenge arises from the difficulty of obtaining comprehensive training data for UGCG images and the unique nature of these images, which differ from traditional unsafe content. In this work, we take the first step towards studying the threat of illicit promotions of unsafe UGCGs. We collect a real-world dataset comprising 2,924 images that display diverse sexually explicit and violent content used to promote UGCGs by their game creators. Our in-depth studies reveal a new understanding of this problem and the urgent need for automatically flagging illicit UGCG promotions. We additionally create a cutting-edge system, UGCG-Guard, designed to aid social media platforms in effectively identifying images used for illicit UGCG promotions. This system leverages recently introduced large vision-language models (VLMs) and employs a novel conditional prompting strategy for zero-shot domain adaptation, along with chain-of-thought (CoT) reasoning for contextual identification. UGCG-Guard achieves outstanding results, with an accuracy rate of 94% in detecting these images used for the illicit promotion of such games in real-world scenarios.