AI Personalization Paradox: Personalized AI Increases Superficial Engagement in Reading while Undermines Autonomy and Ownership in Writing

TL;DR

This study explores how integrating reading highlights into AI-assisted writing affects user behavior, revealing increased reliance on AI and reduced feelings of autonomy and ownership, highlighting risks in personalization strategies.

Contribution

It uniquely examines reading behaviors' role in personalized AI writing, showing how highlighting influences reliance and perceptions of autonomy, which was previously underexplored.

Findings

Highlighting increased AI reliance

Shift from sense-making to feeding AI

Decreased autonomy and ownership

Abstract

AI-assisted writing raises concerns about autonomy and ownership when benefiting writers. Personalization has been proposed as an effective solution while also risking writers' reliance on AI and behavior shifting. For better personalization design, existing studies rely on interaction and information solely within the writing phase; however, few studies have examined how reading behaviors can inform personalized writing. This study investigates the effects of integrating reading highlights for personalization on AI-assisted writing. A between-subjects study with 46 participants revealed that the personalization condition encouraged participants to produce more highlights. However, highlighting unexpectedly shifted from a sense-making strategy to an instrumental act of "feeding the AI," leading to significant reliance on AI and declines in writers' sense of autonomy, ownership, and self-credit. These findings indicate personalization risks in AI-assisted writing, emphasize the importance of personalization strategies, and provide design implications.