CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems

TL;DR

CRS Arena is a new platform for scalable, human-feedback-based benchmarking of conversational recommender systems, enabling reliable evaluation and ranking through pairwise comparisons and crowdsourced data collection.

Contribution

It introduces CRS Arena, a novel benchmarking platform that uses pairwise battles and crowdsourcing to evaluate and rank conversational recommender systems.

Findings

High correlation between open and closed crowdsourcing rankings

Release of CRS Arena dataset with 474 conversations

Preliminary system rankings using Elo rating system

Abstract

We introduce CRS Arena, a research platform for scalable benchmarking of Conversational Recommender Systems (CRS) based on human feedback. The platform displays pairwise battles between anonymous conversational recommender systems, where users interact with the systems one after the other before declaring either a winner or a draw. CRS Arena collects conversations and user feedback, providing a foundation for reliable evaluation and ranking of CRSs. We conduct experiments with CRS Arena on both open and closed crowdsourcing platforms, confirming that both setups produce highly correlated rankings of CRSs and conversations with similar characteristics. We release CRSArena-Dial, a dataset of 474 conversations and their corresponding user feedback, along with a preliminary ranking of the systems based on the Elo rating system. The platform is accessible at https://iai-group-crsarena.hf.space/.