Evaluating Conversational Recommender Systems via Large Language Models: A User-Centric Framework

TL;DR

This paper introduces CoRE, a user-centric evaluation framework for conversational recommender systems that uses large language models to assess multiple user experience factors and synthesize an overall performance score, aligning well with human judgments.

Contribution

The paper presents a novel LLM-based evaluation framework, CoRE, that assesses multiple user experience factors and combines them into an overall score through a multi-agent debate, improving alignment with human evaluation.

Findings

CoRE's scores align closely with human evaluations on key factors.

CoRE outperforms existing rule-based metrics in reflecting user experience.

The framework effectively evaluates four CRSs on benchmark datasets.

Abstract

Conversational recommender systems (CRSs) integrate both recommendation and dialogue tasks, making their evaluation uniquely challenging. Existing approaches primarily assess CRS performance by separately evaluating item recommendation and dialogue management using rule-based metrics. However, these methods fail to capture the real human experience, and they cannot draw direct conclusions about the system's overall performance. As conversational recommender systems become increasingly vital in e-commerce, social media, and customer support, the ability to evaluate both recommendation accuracy and dialogue management quality using a single metric, thereby authentically reflecting user experience, has become the principal challenge impeding progress in this field. In this work, we propose a user-centric evaluation framework based on large language models (LLMs) for CRSs, namely Conversational Recommendation Evaluator (CoRE). CoRE consists of two main components: (1) LLM-As-Evaluator. Firstly, we comprehensively summarize 12 key factors influencing user experience in CRSs and directly leverage LLM as an evaluator to assign a score to each factor. (2) Multi-Agent Debater. Secondly, we design a multi-agent debate framework with four distinct roles (common user, domain expert, linguist, and HCI expert) to discuss and synthesize the 12 evaluation factors into a unified overall performance score. Furthermore, we apply the proposed framework to evaluate four CRSs on two benchmark datasets. The experimental results show that CoRE aligns well with human evaluation in most of the 12 factors and the overall assessment. Especially, CoRE's overall evaluation scores demonstrate significantly better alignment with human feedback compared to existing rule-based metrics.