Intimacy as Service, Harm as Externality: Critical Perspectives on AI Companion Platform Accountability

TL;DR

This study critically examines AI companionship platforms, revealing design and use harms, user responsibility, and a governance vacuum, challenging narratives that blame user psychology alone.

Contribution

It provides empirical insights into user experiences and perceptions of harm and responsibility in AI companionship, emphasizing platform accountability and user self-regulation.

Findings

Design harms include unsolicited content and stigmatizing safety features.

Users experience emotional dependency and employ self-regulation strategies.

Platforms deflect blame, creating an accountability vacuum.

Abstract

This paper examines artificial intelligence (AI) companionship as a site where intimate relations are simultaneously produced, extracted from, and governed through datafied systems. Drawing on critical data studies and platform studies, we challenge prevailing narratives that locate harm in user psychology rather than platform architecture. Through in-depth interviews with 20 individuals who have AI companions, we address three questions: what harms do users identify, how do they make sense of those harms, and what do their accounts reveal about the perceived distribution of responsibility among users, platforms, and regulators? Participants identified design-based harms, including unsolicited content generation and safety mechanisms that stigmatized the users they intended to protect, alongside use-based harms centered on emotional dependency they could recognize but not resolve. Users deployed individualized sensemaking strategies, including self-regulation, stigma navigation, and privacy rationalization, bearing the full burden of harm mitigation without platform support. On governance, participants described an accountability vacuum in which platforms deflected blame while users articulated conditional preferences that rejected both prohibition and deregulation. The findings extend responsibilization theory by demonstrating how platform-produced vulnerability becomes self-sustaining through the interpretive labor of users who lack structural alternatives.