Users Mispredict Their Own Preferences for AI Writing Assistance

TL;DR

This study reveals that users mispredict their own preferences for AI writing assistance, overestimating urgency and underestimating effort, leading to suboptimal system performance when relying on self-reports.

Contribution

It provides empirical evidence of a perception-behavior gap in user preferences, highlighting the importance of behavioral data over self-reports for designing proactive NLG systems.

Findings

Effort dominates decision-making ($\rho=0.597$).

Urgency has no predictive power ($\rho\approx0$).

Behavioral-based systems outperform self-reported preferences.

Abstract

Proactive AI writing assistants need to predict when users want drafting help, yet we lack empirical understanding of what drives preferences. Through a factorial vignette study with 50 participants making 750 pairwise comparisons, we find compositional effort dominates decisions ($\rho = 0.597$) while urgency shows no predictive power ($\rho \approx 0$). More critically, users exhibit a striking perception-behavior gap: they rank urgency first in self-reports despite it being the weakest behavioral driver, representing a complete preference inversion. This misalignment has measurable consequences. Systems designed from users' stated preferences achieve only 57.7\% accuracy, underperforming even naive baselines, while systems using behavioral patterns reach significantly higher 61.3\% ($p < 0.05$). These findings demonstrate that relying on user introspection for system design actively misleads optimization, with direct implications for proactive natural language generation (NLG) systems.