ERASE -- A Real-World Aligned Benchmark for Unlearning in Recommender Systems

TL;DR

ERASE is a comprehensive benchmark designed to evaluate machine unlearning methods in real-world recommender systems, addressing practical constraints and diverse scenarios to advance the field.

Contribution

It introduces ERASE, a large-scale, realistic benchmark covering multiple tasks, datasets, and algorithms, with extensive artifacts for systematic evaluation of unlearning methods.

Findings

Approximate unlearning can match retraining in some cases.

Robustness of unlearning varies across datasets and models.

Recommender-specific methods are more reliable than general-purpose ones.

Abstract

Machine unlearning (MU) enables the removal of selected training data from trained models, to address privacy compliance, security, and liability issues in recommender systems. Existing MU benchmarks poorly reflect real-world recommender settings: they focus primarily on collaborative filtering, assume unrealistically large deletion requests, and overlook practical constraints such as sequential unlearning and efficiency. We present ERASE, a large-scale benchmark for MU in recommender systems designed to align with real-world usage. ERASE spans three core tasks -- collaborative filtering, session-based recommendation, and next-basket recommendation -- and includes unlearning scenarios inspired by real-world applications, such as sequentially removing sensitive interactions or spam. The benchmark covers seven unlearning algorithms, including general-purpose and recommender-specific methods, across nine public datasets and nine state-of-the-art models. We execute ERASE to produce more than 600 GB of reusable artifacts, such as extensive experimental logs and more than a thousand model checkpoints. Crucially, the artifacts that we release enable systematic analysis of where current unlearning methods succeed and where they fall short. ERASE showcases that approximate unlearning can match retraining in some settings, but robustness varies widely across datasets and architectures. Repeated unlearning exposes weaknesses in general-purpose methods, especially for attention-based and recurrent models, while recommender-specific approaches behave more reliably. ERASE provides the empirical foundation to help the community assess, drive, and track progress toward practical MU in recommender systems.