Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems

TL;DR

This paper proposes a more realistic benchmark for machine unlearning in recommender systems, addressing current limitations by including diverse tasks, scenarios, and efficiency constraints, and demonstrates preliminary positive results in sequential recommendation settings.

Contribution

It introduces a set of design criteria and research questions for developing a comprehensive, real-world unlearning benchmark for recommender systems, expanding beyond existing narrow evaluations.

Findings

Unlearning is feasible for sequential recommendation models.

A custom unlearning algorithm outperforms general methods.

Unlearning can be performed with latency of a few seconds.

Abstract

Modern recommender systems heavily leverage user interaction data to deliver personalized experiences. However, relying on personal data presents challenges in adhering to privacy regulations, such as the GDPR's "right to be forgotten". Machine unlearning (MU) aims to address these challenges by enabling the efficient removal of specific training data from models post-training, without compromising model utility or leaving residual information. However, current benchmarks for unlearning in recommender systems -- most notably CURE4Rec -- fail to reflect real-world operational demands. They focus narrowly on collaborative filtering, overlook tasks like session-based and next-basket recommendation, simulate unrealistically large unlearning requests, and ignore critical efficiency constraints. In this paper, we propose a set of design desiderata and research questions to guide the development of a more realistic benchmark for unlearning in recommender systems, with the goal of gathering feedback from the research community. Our benchmark proposal spans multiple recommendation tasks, includes domain-specific unlearning scenarios, and several unlearning algorithms -- including ones adapted from a recent NeurIPS unlearning competition. Furthermore, we argue for an unlearning setup that reflects the sequential, time-sensitive nature of real-world deletion requests. We also present a preliminary experiment in a next-basket recommendation setting based on our proposed desiderata and find that unlearning also works for sequential recommendation models, exposed to many small unlearning requests. In this case, we observe that a modification of a custom-designed unlearning algorithm for recommender systems outperforms general unlearning algorithms significantly, and that unlearning can be executed with a latency of only several seconds.