UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems

TL;DR

UserSimCRS is an extensible toolkit that automates the evaluation of conversational recommender systems by simulating realistic user interactions using novel modeling techniques.

Contribution

It introduces new features like user satisfaction prediction, persona and context modeling, and natural language generation to improve simulation realism.

Findings

Successfully simulates realistic dialogues with minimal training data

Enhances evaluation accuracy for conversational recommender systems

Demonstrates effectiveness with a movie recommender system

Abstract

We present an extensible user simulation toolkit to facilitate automatic evaluation of conversational recommender systems. It builds on an established agenda-based approach and extends it with several novel elements, including user satisfaction prediction, persona and context modeling, and conditional natural language generation. We showcase the toolkit with a pre-existing movie recommender system and demonstrate its ability to simulate dialogues that mimic real conversations, while requiring only a handful of manually annotated dialogues as training data.