Evaluation on Entity Matching in Recommender Systems

TL;DR

This paper introduces a new dataset for cross-dataset entity matching in recommender systems, evaluates various matching methods, and provides a gold standard to facilitate future research in this area.

Contribution

It presents Reddit-Amazon-EM, a manually annotated dataset for entity matching, and offers a comprehensive evaluation of multiple matching techniques including LLM-based approaches.

Findings

LLM-based methods perform competitively in entity matching.

The dataset enables reproducible evaluation of matching algorithms.

The best method achieves high accuracy in cross-dataset entity alignment.

Abstract

Entity matching is a crucial component in various recommender systems, including conversational recommender systems (CRS) and knowledge-based recommender systems. However, the lack of rigorous evaluation frameworks for cross-dataset entity matching impedes progress in areas such as LLM-driven conversational recommendations and knowledge-grounded dataset construction. In this paper, we introduce Reddit-Amazon-EM, a novel dataset comprising naturally occurring items from Reddit and the Amazon '23 dataset. Through careful manual annotation, we identify corresponding movies across Reddit-Movies and Amazon'23, two existing recommender system datasets with inherently overlapping catalogs. Leveraging Reddit-Amazon-EM, we conduct a comprehensive evaluation of state-of-the-art entity matching methods, including rule-based, graph-based, lexical-based, embedding-based, and LLM-based approaches. For reproducible research, we release our manually annotated entity matching gold set and provide the mapping between the two datasets using the best-performing method from our experiments. This serves as a valuable resource for advancing future work on entity matching in recommender systems.Data and Code are accessible at: https://github.com/huang-zihan/Reddit-Amazon-Entity-Matching.