Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm

TL;DR

This paper analyzes how personalization in open-source text-to-image models can perpetuate misogyny and harmful content, highlighting sociotechnical factors and proposing interventions for safer AI practices.

Contribution

It provides an exploratory sociotechnical analysis of CivitAI, revealing how model personalization fosters misogynistic and NSFW content, and suggests strategies for harm mitigation.

Findings

Rise in NSFW content on CivitAI

Prevalence of models mimicking real individuals

Influence of internet subcultures on model practices

Abstract

Open-source text-to-image (TTI) pipelines have become dominant in the landscape of AI-generated visual content, driven by technological advances that enable users to personalize models through adapters tailored to specific tasks. While personalization methods such as LoRA offer unprecedented creative opportunities, they also facilitate harmful practices, including the generation of non-consensual deepfakes and the amplification of misogynistic or hypersexualized content. This study presents an exploratory sociotechnical analysis of CivitAI, the most active platform for sharing and developing open-source TTI models. Drawing on a dataset of more than 40 million user-generated images and over 230,000 models, we find a disproportionate rise in not-safe-for-work (NSFW) content and a significant number of models intended to mimic real individuals. We also observe a strong influence of internet subcultures on the tools and practices shaping model personalizations and resulting visual media. In response to these findings, we contextualize the emergence of exploitative visual media through feminist and constructivist perspectives on technology, emphasizing how design choices and community dynamics shape platform outcomes. Building on this analysis, we propose interventions aimed at mitigating downstream harm, including improved content moderation, rethinking tool design, and establishing clearer platform policies to promote accountability and consent.