Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies

TL;DR

This paper introduces a generative agent-based simulation platform that models human-assistant interactions, enabling scalable, cost-effective research that closely replicates real human studies without involving live participants.

Contribution

The paper presents a novel simulation platform using large language models to emulate human behavior in assistant interactions, reducing reliance on human-in-the-loop experiments.

Findings

Simulated experiments closely match key aspects of human-assistant interactions.

The platform successfully replicates core conclusions of previous human studies.

Provides a scalable, ethical alternative for assistant agent research.

Abstract

Designing and evaluating personalized and proactive assistant agents remains challenging due to the time, cost, and ethical concerns associated with human-in-the-loop experimentation. Existing Human-Computer Interaction (HCI) methods often require extensive physical setup and human participation, which introduces privacy concerns and limits scalability. Simulated environments offer a partial solution but are typically constrained by rule-based scenarios and still depend heavily on human input to guide interactions and interpret results. Recent advances in large language models (LLMs) have introduced the possibility of generative agents that can simulate realistic human behavior, reasoning, and social dynamics. However, their effectiveness in modeling human-assistant interactions remains largely unexplored. To address this gap, we present a generative agent-based simulation platform designed to simulate human-assistant interactions. We identify ten prior studies on assistant agents that span different aspects of interaction design and replicate these studies using our simulation platform. Our results show that fully simulated experiments using generative agents can approximate key aspects of human-assistant interactions. Based on these simulations, we are able to replicate the core conclusions of the original studies. Our work provides a scalable and cost-effective approach for studying assistant agent design without requiring live human subjects. Additional resources and project materials are available at https://dash-gidea.github.io/