Investigating the Robustness of Sequential Recommender Systems Against Training Data Perturbations

TL;DR

This paper introduces a new similarity measure called Finite Rank-Biased Overlap to better evaluate the robustness of sequential recommender systems against data perturbations, revealing that perturbations at sequence ends significantly impact performance.

Contribution

The paper proposes the Finite Rank-Biased Overlap similarity measure and empirically investigates how item removal at different sequence positions affects recommender system robustness.

Findings

Removing items at sequence ends significantly reduces NDCG by up to 60%.

Removing items from the beginning or middle has negligible impact.

The new similarity measure improves evaluation of finite ranking robustness.

Abstract

Sequential Recommender Systems (SRSs) are widely employed to model user behavior over time. However, their robustness in the face of perturbations in training data remains a largely understudied yet critical issue. A fundamental challenge emerges in previous studies aimed at assessing the robustness of SRSs: the Rank-Biased Overlap (RBO) similarity is not particularly suited for this task as it is designed for infinite rankings of items and thus shows limitations in real-world scenarios. For instance, it fails to achieve a perfect score of 1 for two identical finite-length rankings. To address this challenge, we introduce a novel contribution: Finite Rank-Biased Overlap (FRBO), an enhanced similarity tailored explicitly for finite rankings. This innovation facilitates a more intuitive evaluation in practical settings. In pursuit of our goal, we empirically investigate the impact of removing items at different positions within a temporally ordered sequence. We evaluate two distinct SRS models across multiple datasets, measuring their performance using metrics such as Normalized Discounted Cumulative Gain (NDCG) and Rank List Sensitivity. Our results demonstrate that removing items at the end of the sequence has a statistically significant impact on performance, with NDCG decreasing up to 60%. Conversely, removing items from the beginning or middle has no significant effect. These findings underscore the criticality of the position of perturbed items in the training data. As we spotlight the vulnerabilities inherent in current SRSs, we fervently advocate for intensified research efforts to fortify their robustness against adversarial perturbations.