Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations

TL;DR

This paper evaluates the strength of sequential patterns in 15 datasets used for testing sequential recommender systems by comparing original data with shuffled versions to assess the importance of sequence information.

Contribution

It introduces a method to quantify the sequential structure in datasets by shuffling interactions and analyzing the impact on recommendation metrics.

Findings

Several popular datasets have weak sequential structure.

Shuffling reduces the effectiveness of sequential models on some datasets.

The study questions the assumption that all datasets are suitable for evaluating sequential recommenders.

Abstract

Sequential recommender systems are an important and demanded area of research. Such systems aim to use the order of interactions in a user's history to predict future interactions. The premise is that the order of interactions and sequential patterns play an essential role. Therefore, it is crucial to use datasets that exhibit a sequential structure to evaluate sequential recommenders properly. We apply several methods based on the random shuffling of the user's sequence of interactions to assess the strength of sequential structure across 15 datasets, frequently used for sequential recommender systems evaluation in recent research papers presented at top-tier conferences. As shuffling explicitly breaks sequential dependencies inherent in datasets, we estimate the strength of sequential patterns by comparing metrics for shuffled and original versions of the dataset. Our findings show that several popular datasets have a rather weak sequential structure.