Interview-Informed Generative Agents for Product Discovery: A Validation Study

TL;DR

This study evaluates the use of interview-informed generative agents to simulate user responses for product concept testing, revealing their potential for early-stage screening but limitations in individual-level accuracy.

Contribution

It introduces a method for creating personalized agents based on workflow interviews and assesses their effectiveness in simulating user responses in design research.

Findings

Agents are distribution-calibrated but not identity-precise.

They approximate population-level response distributions.

Suitable for early-stage concept screening, not individual insights.

Abstract

Large language models (LLMs) have shown strong performance on standardized social science instruments, but their value for product discovery remains unclear. We investigate whether interview-informed generative agents can simulate user responses in concept testing scenarios. Using in-depth workflow interviews with knowledge workers, we created personalized agents and compared their evaluations of novel AI concepts against the same participants' responses. Our results show that agents are distribution-calibrated but identity-imprecise: they fail to replicate the specific individual they are grounded in, yet approximate population-level response distributions. These findings highlight both the potential and the limits of LLM simulation in design research. While unsuitable as a substitute for individual-level insights, simulation may provide value for early-stage concept screening and iteration, where distributional accuracy suffices. We discuss implications for integrating simulation responsibly into product development workflows.