RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systems

TL;DR

RecUserSim is a novel user simulator leveraging large language models, designed to produce realistic, diverse, and controllable multi-turn dialogues with explicit ratings, significantly improving the evaluation of conversational recommender systems.

Contribution

It introduces RecUserSim, a user simulator with modules for user profiles, memory, decision-making based on Bounded Rationality, and response refinement, enhancing realism and diversity in CRS evaluation.

Findings

RecUserSim generates diverse and realistic dialogues.

It produces high-quality responses even with smaller LLMs.

Ratings are consistent across different base LLMs.

Abstract

Conversational recommender systems (CRS) enhance user experience through multi-turn interactions, yet evaluating CRS remains challenging. User simulators can provide comprehensive evaluations through interactions with CRS, but building realistic and diverse simulators is difficult. While recent work leverages large language models (LLMs) to simulate user interactions, they still fall short in emulating individual real users across diverse scenarios and lack explicit rating mechanisms for quantitative evaluation. To address these gaps, we propose RecUserSim, an LLM agent-based user simulator with enhanced simulation realism and diversity while providing explicit scores. RecUserSim features several key modules: a profile module for defining realistic and diverse user personas, a memory module for tracking interaction history and discovering unknown preferences, and a core action module inspired by Bounded Rationality theory that enables nuanced decision-making while generating more fine-grained actions and personalized responses. To further enhance output control, a refinement module is designed to fine-tune final responses. Experiments demonstrate that RecUserSim generates diverse, controllable outputs and produces realistic, high-quality dialogues, even with smaller base LLMs. The ratings generated by RecUserSim show high consistency across different base LLMs, highlighting its effectiveness for CRS evaluation.