Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studies

TL;DR

This paper presents a scalable method using crowdsourced generative user agents from large profile datasets to conduct cost-effective, large-scale user experience studies that produce insights comparable to traditional methods.

Contribution

It introduces a novel crowdsourcing simulated user agents approach, validated through a game prototyping study, and provides an open-source toolkit for practical adoption in UX research.

Findings

Scaling simulated agents increases coverage of human findings up to 90%.

12.8 simulated agents are as effective as one local human participant.

Aggregated simulated insights are comparable to real user data.

Abstract

User studies are central to user experience research, yet recruiting participant is expensive, slow, and limited in diversity. Recent work has explored using Large Language Models as simulated users, but doubts about fidelity have hindered practical adoption. We deepen this line of research by asking whether scale itself can enable useful simulation, even if not perfectly accurate. We introduce Crowdsourcing Simulated User Agents, a method that recruits generative agents from billion-scale profile assets to act as study participants. Unlike handcrafted simulations, agents are treated as recruitable, screenable, and engageable across UX research stages. To ground this method, we demonstrate a game prototyping study with hundreds of simulated players, comparing their insights against a 10-participant local user study and a 20-participant crowdsourcing study with humans. We find a clear scaling effect: as the number of simulated user agents increases, coverage of human findings rises smoothly and plateaus around 90\%. 12.8 simulated agents are as useful as one locally recruited human, and 3.2 agents are as useful as one crowdsourced human. Results show that while individual agents are imperfect, aggregated simulations produce representative and actionable insights comparable to real users. Professional designers further rated these insights as balancing fidelity, cost, time efficiency, and usefulness. Finally, we release an agent crowdsourcing toolkit with a modular open-source pipeline and a curated pool of profiles synced from ongoing simulation research, to lower the barrier for researchers to adopt simulated participants. Together, this work contributes a validated method and reusable toolkit that expand the options for conducting scalable and practical UX studies.