INSPIRED2: An Improved Dataset for Sociable Conversational Recommendation

TL;DR

This paper introduces INSPIRED2, an improved version of a sociable conversational recommendation dataset, with manually corrected annotations that enhance the performance of benchmark CRS models, highlighting the importance of data quality.

Contribution

The paper presents a manually corrected version of the INSPIRED dataset, called INSPIRED2, demonstrating its positive impact on CRS performance and emphasizing the significance of annotation accuracy.

Findings

INSPIRED2 improves CRS performance over the original dataset.

Manual annotation correction reduces errors and missing data.

Enhanced data quality benefits both end-to-end and retrieval-based CRS models.

Abstract

Conversational recommender systems (CRS) that are able to interact with users in natural language often utilize recommendation dialogs which were previously collected with the help of paired humans, where one plays the role of a seeker and the other as a recommender. These recommendation dialogs include items and entities that indicate the users' preferences. In order to precisely model the seekers' preferences and respond consistently, CRS typically rely on item and entity annotations. A recent example of such a dataset is INSPIRED, which consists of recommendation dialogs for sociable conversational recommendation, where items and entities were annotated using automatic keyword or pattern matching techniques. An analysis of this dataset unfortunately revealed that there is a substantial number of cases where items and entities were either wrongly annotated or annotations were missing at all. This leads to the question to what extent automatic techniques for annotations are effective. Moreover, it is important to study impact of annotation quality on the overall effectiveness of a CRS in terms of the quality of the system's responses. To study these aspects, we manually fixed the annotations in INSPIRED. We then evaluated the performance of several benchmark CRS using both versions of the dataset. Our analyses suggest that the improved version of the dataset, i.e., INSPIRED2, helped increase the performance of several benchmark CRS, emphasizing the importance of data quality both for end-to-end learning and retrieval-based approaches to conversational recommendation. We release our improved dataset (INSPIRED2) publicly at https://github.com/ahtsham58/INSPIRED2.