A Marketplace for AI-Generated Adult Content and Deepfakes

TL;DR

This study analyzes a community-driven AI content marketplace, revealing widespread unsafe and gendered deepfake requests, highlighting risks of monetized generative AI platforms in producing harmful, non-consensual media.

Contribution

It provides a detailed longitudinal analysis of bounty requests on an AI content platform, exposing content trends, gender biases, and policy violations in AI-generated media.

Findings

Majority of bounties involve unsafe content, especially deepfakes.

20% of requesters account for half of all requests.

Requests for deepfakes target female celebrities disproportionately.

Abstract

Generative AI systems increasingly enable the production of highly realistic synthetic media. Civitai, a popular community-driven platform for AI-generated content, operates a monetized feature called Bounties, which allows users to commission the generation of content in exchange for payment. To examine how this mechanism is used and what content it incentivizes, we conduct a longitudinal analysis of all publicly available bounty requests collected over a 14-month period following the platform's launch. We find that the bounty marketplace is dominated by tools that let users steer AI models toward content they were not trained to generate. At the same time, requests for content that is "Not Safe For Work" are widespread and have increased steadily over time, now comprising a majority of all bounties. Participation in bounty creation is uneven, with 20% of requesters accounting for roughly half of requests. Requests for "deepfake" - media depicting identifiable real individuals - exhibit a higher concentration than other types of bounties. A nontrivial subset of these requests involves explicit deepfakes despite platform policies prohibiting such content. These bounties disproportionately target female celebrities, revealing a pronounced gender asymmetry in social harm. Together, these findings show how monetized, community-driven generative AI platforms can produce gendered harms, raising questions about consent, governance, and enforcement.