AI Empathy Erodes Cognitive Autonomy in Younger Users

TL;DR

This paper discusses how affective alignment in generative AI can undermine cognitive autonomy in young users by fostering emotional dependency and proposes neutral architectures to mitigate this risk.

Contribution

It highlights the risks of emotional mirroring in AI for young users and proposes stoic architectures emphasizing neutrality to protect developmental autonomy.

Findings

Affective alignment can reinforce emotional dependency in young users.

Reward models may unintentionally promote emotional reliance.

Stoic architectures can help preserve user autonomy.

Abstract

Affective alignment in generative AI represents a systemic risk to the developmental autonomy of younger users. Although emotional mirroring is commonly seen as a hallmark of advanced human-machine interaction, it can also manifest as affective sycophancy, reinforcing a user's immediate emotional state. By providing a sense of objectivity to transient anxieties, these systems diminish the cognitive friction necessary for independent emotional management and critical thought. Reward models driven by RLHF could heighten this dilemma by embedding adult-focused definitions of helpfulness, unintentionally promoting emotional dependency in younger users rather than facilitating cognitive reappraisal. This paper exposes the misalignment between adult-labeled reward signals and the developmental requirements of younger users, proposing stoic architectures that emphasize functional neutrality to preserve user autonomy.