Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems

TL;DR

Hesitator is a decision-aware user simulation framework for evaluating conversational recommender systems, explicitly modeling human decision-making and overload effects to produce more realistic user behaviors.

Contribution

It introduces a modular decision module that separates utility-based choices from overload-aware commitments, improving realism in user simulations.

Findings

Mitigates unrealistic behaviors under overload conditions

Reproduces established behavioral patterns from psychology economics

Consistently improves simulation realism across domains and models

Abstract

Conversational recommender systems (CRS) increasingly rely on user simulators for automated evaluation of sales agents. A key requirement for such simulators is the ability to model human decision-making. However, most existing simulation frameworks do not explicitly model the internal decision process, and LLM-based simulators often exhibit unrealistically strong information-processing capabilities, rarely exhibit the hesitation or decision deferral commonly observed in real consumer behavior, resulting in overly high acceptance probabilities. To address this limitation, we propose Hesitator, a theory-grounded user simulation framework that explicitly models human decision-making under choice overload. The framework introduces a modular Decision Module that separates utility-based item selection from overload-aware commitment decisions. Experiments across multiple user simulation frameworks, domains, sales modes, and LLM backbones show that integrating our module consistently mitigates unrealistic behaviors under increasing overload conditions. Furthermore, Hesitator reproduces established behavioral patterns from psychological economics, demonstrating its ability to model human decision behavior.