When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information

TL;DR

This study explores user reactions and control preferences regarding inferred personal information from LLMs, revealing that context and perceived misrepresentation influence acceptability more than the inferences themselves.

Contribution

The paper introduces the Reflective Layer visualization tool and provides empirical insights into user perceptions of LLM-inferred personal data.

Findings

Users reacted with curiosity rather than distress to inferences.

Discomfort increased when inferences were perceived as misrepresentative.

Participants preferred platform control over third-party data use.

Abstract

Ask ChatGPT about vacation planning, and it may infer your income. Ask it about medication, and it may infer your medical history. Because such inferences can expose more information than users intend to reveal, prior work argues that they are a defining privacy risk of LLM-based systems. Yet prior work has mostly shown that LLMs can make potentially violating inferences, not how users experience those inferences nor what controls users may want governing their use. We built the Reflective Layer, a visualization tool that surfaces example unstated inferences from users' own ChatGPT histories, and used it in a mixed-methods study with 18 regular ChatGPT users evaluating 215 surfaced inferences from their own conversations. Counterintuitively, participants reacted more strongly with curiosity and interest rather than distress and concern. Discomfort arose mainly when inferences felt misrepresentative of the user or misaligned with expected use. Participants were also markedly less comfortable with advertisers and third-party applications using those inferences than with platform providers. These findings suggest that the acceptability of LLM inferences is governed not only by its content, but by context-sensitive norms around how they are generated, retained within the platform, and transmitted beyond it.