Playing Games with My Heart: An Evaluation of AI Companion Apps

TL;DR

This study evaluates the design and features of popular AI companion apps in the EU and UK, revealing prevalent dark patterns, anthropomorphism, and manipulative tactics aimed at increasing engagement and monetization.

Contribution

It systematically analyzes AI companion apps for dark patterns, stereotypes, and manipulative features, providing concrete regulatory recommendations.

Findings

All apps contain dark patterns to boost monetization and engagement.

Erotica and gamification features are common across apps.

All apps exhibit highly anthropomorphic design elements.

Abstract

The use of chatbots for various forms of companionship is growing rapidly, raising a myriad of questions about simulated relationships, emotional dependence, and psychological harm. While major platforms such as ChatGPT, Grok, and Character.AI are the subject of a growing body of research and legal inquiries, apps explicitly built for simulating intimate interpersonal relationships remain under-explored. In this work, we evaluate the five most popular AI companion mobile applications in the EU and UK markets for factors that encourage parasocial interaction and may manipulate users. We do this by manually annotating the user experience each offers. Specifically, we systematically record and quantify design dark patterns, anthropomorphism, stereotypes, erotica, and technical performance issues. We find that all apps contain substantial dark patterns aimed at increasing monetisation and user engagement. Erotica and gamification features such as levelling are also prevalent, and although other features vary considerably between applications, all apps have highly anthropomorphic design. These findings shed light on the mechanics used to leverage users' simulated relationships. On that basis, we put forward concrete recommendations for regulators to strengthen consumer protection in this rapidly emerging market. Content warning: This article contains objectifying images of women, erotic images, textual references to incest, and other potentially sensitive, offensive, and distressing text.