Evaluating User Experience in Conversational Recommender Systems: A Systematic Review Across Classical and LLM-Powered Approaches

TL;DR

This systematic review examines how user experience in conversational recommender systems is evaluated, highlighting gaps in empirical studies, especially for LLM-based systems, and proposing a structured framework for better UX assessment.

Contribution

It provides a comprehensive synthesis of UX evaluation methods in CRSs, compares adaptive and nonadaptive systems, and offers a future agenda for LLM-aware UX evaluation.

Findings

Post hoc surveys dominate UX evaluation methods.

Turn-level affective UX is rarely assessed.

LLM-based CRSs face challenges like opacity and verbosity.

Abstract

Conversational Recommender Systems (CRSs) are receiving growing research attention across domains, yet their user experience (UX) evaluation remains limited. Existing reviews largely overlook empirical UX studies, particularly in adaptive and large language model (LLM)-based CRSs. To address this gap, we conducted a systematic review following PRISMA guidelines, synthesising 23 empirical studies published between 2017 and 2025. We analysed how UX has been conceptualised, measured, and shaped by domain, adaptivity, and LLM. Our findings reveal persistent limitations: post hoc surveys dominate, turn-level affective UX constructs are rarely assessed, and adaptive behaviours are seldom linked to UX outcomes. LLM-based CRSs introduce further challenges, including epistemic opacity and verbosity, yet evaluations infrequently address these issues. We contribute a structured synthesis of UX metrics, a comparative analysis of adaptive and nonadaptive systems, and a forward-looking agenda for LLM-aware UX evaluation. These findings support the development of more transparent, engaging, and user-centred CRS evaluation practices.