"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents

TL;DR

This paper investigates how users perceive and navigate privacy risks when using LLM-based conversational agents, revealing misconceptions and design issues that affect privacy awareness and decision-making.

Contribution

It provides the first user-centered analysis of privacy concerns and behaviors in real-world LLM-based chat interactions, highlighting mental models and system design impacts.

Findings

Users face trade-offs between privacy, utility, and convenience.

Erroneous mental models limit privacy risk awareness.

Human-like interactions increase sensitive disclosures.

Abstract

The widespread use of Large Language Model (LLM)-based conversational agents (CAs), especially in high-stakes domains, raises many privacy concerns. Building ethical LLM-based CAs that respect user privacy requires an in-depth understanding of the privacy risks that concern users the most. However, existing research, primarily model-centered, does not provide insight into users' perspectives. To bridge this gap, we analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs. However, users' erroneous mental models and the dark patterns in system design limited their awareness and comprehension of the privacy risks. Additionally, the human-like interactions encouraged more sensitive disclosures, which complicated users' ability to navigate the trade-offs. We discuss practical design guidelines and the needs for paradigm shifts to protect the privacy of LLM-based CA users.