The Impostor is Among Us: Can Large Language Models Capture the Complexity of Human Personas?

TL;DR

This study compares human and AI-generated personas, revealing that while LLMs produce informative and consistent personas, they often rely on stereotypes, underscoring challenges in capturing human complexity for design.

Contribution

The paper systematically evaluates the differences between human-crafted and LLM-generated personas, highlighting strengths and limitations of AI in representing human diversity.

Findings

Participants perceived AI personas as more informative and consistent.

AI personas tend to follow stereotypes, reducing diversity.

Users can distinguish between human and AI-generated personas.

Abstract

Large Language Models (LLMs) created new opportunities for generating personas, expected to streamline and accelerate the human-centered design process. Yet, AI-generated personas may not accurately represent actual user experiences, as they can miss contextual and emotional insights critical to understanding real users' needs and behaviors. This introduces a potential threat to quality, especially for novices. This paper examines the differences in how users perceive personas created by LLMs compared to those crafted by humans regarding their credibility for design. We gathered ten human-crafted personas developed by HCI experts according to relevant attributes established in related work. Then, we systematically generated ten personas with an LLM and compared them with human-crafted ones in a survey. The results showed that participants differentiated between human-created and AI-generated personas, with the latter perceived as more informative and consistent. However, participants noted that the AI-generated personas tended to follow stereotypes, highlighting the need for a greater emphasis on diversity when utilizing LLMs for persona creation.