Examining Risks in the AI Companion Application Ecosystem

TL;DR

This paper analyzes the security risks in AI companion apps, identifying threats to users and malicious exploitation, through systematic analysis of 30 apps and ecosystem trends, highlighting issues like data privacy, anthropomorphism, and synthetic media risks.

Contribution

It provides a comprehensive threat model and ecosystem analysis for AI companion applications, highlighting specific risks and informing future security, policy, and technical developments.

Findings

Identified 489 AI companion apps in app stores.

Analyzed 30 apps to understand threat categories.

Highlighted risks related to data sharing, synthetic media, and user manipulation.

Abstract

While computer systems that allow users to interact through conversational natural language (i.e., chatbots) have existed for many years, varying types of applications advertising AI companionship (e.g., Character AI, Replika) have proliferated in recent years due to advancements in large language models. Our work offers a threat model encompassing two distinct risk categories: harms posed to users by AI companion applications, and harms enabled by malicious users exploiting application features. To further understand this application ecosystem, we identified 489 unique apps from the App Store and Play Store that advertised AI companionship. We then systematically conducted and analyzed walkthroughs of a stratified sample of 30 apps with respect to our threat model. Through our analysis, we categorize broader ecosystem trends that provide context for understanding threats and identify specific threats related to sensitive data collection and sharing, anthropomorphism, engagement mechanisms, sexual interactions and media, as well as the ingestion and reconstruction of likeness, including the potential for generating synthetic nonconsensual intimate imagery. This study provides a foundational security perspective on the AI companion application ecosystem and informs future research within and beyond this field, policy, and technical development. Content warning: This paper includes descriptions of applications that can be used to create synthetic nonconsensual representations, including explicit imagery, as well as discussion of self-harm and suicidal ideation.