Testing Deep Learning Recommender Systems Models on Synthetic GAN-Generated Datasets

TL;DR

This paper evaluates the GANRS method for generating synthetic datasets for recommender systems, testing its effectiveness across various models and dataset configurations, and analyzing the consistency of synthetic data with real data in performance metrics.

Contribution

It provides an empirical assessment of GANRS-generated datasets' quality and their suitability for benchmarking deep learning recommender models.

Findings

Synthetic datasets show consistent performance trends with real data.

Deep learning models perform similarly on synthetic and real datasets.

GANRS-generated data can be used for comparative evaluation of recommender systems.

Abstract

The published method Generative Adversarial Networks for Recommender Systems (GANRS) allows generating data sets for collaborative filtering recommendation systems. The GANRS source code is available along with a representative set of generated datasets. We have tested the GANRS method by creating multiple synthetic datasets from three different real datasets taken as a source. Experiments include variations in the number of users in the synthetic datasets, as well as a different number of samples. We have also selected six state-of-the-art collaborative filtering deep learning models to test both their comparative performance and the GANRS method. The results show a consistent behavior of the generated datasets compared to the source ones; particularly, in the obtained values and trends of the precision and recall quality measures. The tested deep learning models have also performed as expected on all synthetic datasets, making it possible to compare the results with those obtained from the real source data. Future work is proposed, including different cold start scenarios, unbalanced data, and demographic fairness.