Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items

TL;DR

Fashion-AlterEval introduces a new dataset with human judgments on alternative items, enabling more realistic evaluation of conversational recommendation systems and revealing that considering alternatives improves system performance assessment.

Contribution

The paper presents Fashion-AlterEval, a novel dataset with annotations for alternative items and two meta-user simulators that model user behavior more realistically in CRS evaluations.

Findings

Alternative-aware evaluation shows higher CRS effectiveness.

Simulated users' ability to change preferences impacts system assessment.

Considering alternatives leads to faster user satisfaction in simulations.

Abstract

In Conversational Recommendation Systems (CRS), a user provides feedback on recommended items at each turn, leading the CRS towards improved recommendations. Due to the need for a large amount of data, a user simulator is employed for both training and evaluation. Such user simulators critique the current retrieved item based on knowledge of a single target item. However, system evaluation in offline settings with simulators is limited by the focus on a single target item and their unlimited patience over a large number of turns. To overcome these limitations of existing simulators, we propose Fashion-AlterEval, a new dataset that contains human judgments for a selection of alternative items by adding new annotations in common fashion CRS datasets. Consequently, we propose two novel meta-user simulators that use the collected judgments and allow simulated users not only to express their preferences about alternative items to their original target, but also to change their mind and level of patience. In our experiments using the Shoes and Fashion IQ as the original datasets and three CRS models, we find that using the knowledge of alternatives by the simulator can have a considerable impact on the evaluation of existing CRS models, specifically that the existing single-target evaluation underestimates their effectiveness, and when simulatedusers are allowed to instead consider alternative relevant items, the system can rapidly respond to more quickly satisfy the user.