Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts

TL;DR

This paper explores user perceptions of system prompts in generative AI, highlighting their influence, transparency issues, and implications for design and governance to better align AI behavior with user expectations.

Contribution

It provides empirical insights into user perspectives on system prompts, emphasizing the need for transparent, user-centered design practices in LLM systems.

Findings

Users see benefits and risks in system prompts

Transparency and control over prompts vary in user perceptions

Design considerations can improve alignment with user values

Abstract

System prompts - instructions that shape the behaviour of generative AI systems - strongly influence system outputs and users' experiences. They define the model's guidelines, `personality', and guardrails, taking precedence over user inputs. Despite their influence, transparency is limited: system prompts are generally not made public and most platforms instruct models to conceal them, leaving users disconnected from and unaware of a key mechanism guiding and governing their AI interactions. This paper argues that system prompts warrant explicit, user-centred design attention and, focusing on large language models (LLMs), asks: what do system prompts contain, how do end-users perceive them, and what do these perceptions offer for design and governance practice? Our results reveal user perspectives on: the benefits and risks of system prompts; the values they prefer to be associated with prompt-design; their levels of comfort with different types of prompts; and degrees of transparency and user control regarding prompt content. From these findings emerge considerations for how designers can better align system prompt mechanisms with user expectations and preferences over these mechanisms that directly shape how generative AI systems behave.