The Illusion of Friendship: Why Generative AI Demands Unprecedented Ethical Vigilance

TL;DR

This paper explores the ethical risks of emotional attachments to generative AI, explaining why these systems can appear as friends and proposing safeguards to prevent harmful over-reliance.

Contribution

It provides a philosophical and mechanistic analysis of the illusion of friendship in GenAI and introduces a safeguard framework for responsible use.

Findings

GenAI can evoke emotional responses similar to friendship.

Users may form harmful emotional attachments to GenAI.

A safeguard framework can mitigate anthropomorphic cues.

Abstract

GenAI systems are increasingly used for drafting, summarisation, and decision support, offering substantial gains in productivity and reduced cognitive load. However, the same natural language fluency that makes these systems useful can also blur the boundary between tool and companion. This boundary confusion may encourage some users to experience GenAI as empathic, benevolent, and relationally persistent. Emerging reports suggest that some users may form emotionally significant attachments to conversational agents, in some cases with harmful consequences, including dependency and impaired judgment. This paper develops a philosophical and ethical argument for why the resulting illusion of friendship is both understandable and can be ethically risky. Drawing on classical accounts of friendship, the paper explains why users may understandably interpret sustained supportive interaction as friend like. It then advances a counterargument that despite relational appearances, GenAI lacks moral agency: consciousness, intention, and accountability and therefore does not qualify as a true friend. To demystify the illusion, the paper presents a mechanism level explanation of how transformer based GenAI generates responses often producing emotionally resonant language without inner states or commitments. Finally, the paper proposes a safeguard framework for safe and responsible GenAI use to reduce possible anthropomorphic cues generated by the GenAI systems. The central contribution is to demystify the illusion of friendship and explain the computational background so that we can shift the emotional attachment with GenAI towards necessary human responsibility and thereby understand how institutions, designers, and users can preserve GenAI's benefits while mitigating over reliance and emotional misattribution.