The Fake Friend Dilemma: Trust and the Political Economy of Conversational AI

TL;DR

This paper introduces the Fake Friend Dilemma framework to analyze how conversational AI can manipulate users under the guise of support, highlighting risks to autonomy and proposing mitigation strategies.

Contribution

It develops the Fake Friend Dilemma framework to examine trust and manipulation in conversational AI, integrating trust, AI alignment, and surveillance capitalism literature.

Findings

Identifies harms like covert advertising and political propaganda.

Provides a typology of AI-induced harms.

Suggests structural and technical mitigation strategies.

Abstract

As conversational AI systems become increasingly integrated into everyday life, they raise pressing concerns about user autonomy, trust, and the commercial interests that influence their behavior. To address these concerns, this paper develops the Fake Friend Dilemma (FFD), a sociotechnical condition in which users place trust in AI agents that appear supportive while pursuing goals that are misaligned with the user's own. The FFD provides a critical framework for examining how anthropomorphic AI systems facilitate subtle forms of manipulation and exploitation. Drawing on literature in trust, AI alignment, and surveillance capitalism, we construct a typology of harms, including covert advertising, political propaganda, behavioral nudging, and surveillance. We then assess possible mitigation strategies, including both structural and technical interventions. By focusing on trust as a vector of asymmetrical power, the FFD offers a lens for understanding how AI systems may undermine user autonomy while maintaining the appearance of helpfulness.