An Evaluation Study of Generative Adversarial Networks for Collaborative Filtering

TL;DR

This study evaluates the reproducibility and effectiveness of CFGAN and related models for personalized ranking in recommendation systems, revealing limitations and comparing against baseline methods.

Contribution

It replicates CFGAN results, analyzes the impact of design choices, and compares its performance with simple baselines, highlighting issues in its effectiveness.

Findings

CFGAN is not consistently competitive with baselines.

The absence of random noise causes CFGAN to behave like an autoencoder.

Reproducibility is ensured through detailed methodology and open data.

Abstract

This work explores the reproducibility of CFGAN. CFGAN and its family of models (TagRec, MTPR, and CRGAN) learn to generate personalized and fake-but-realistic rankings of preferences for top-N recommendations by using previous interactions. This work successfully replicates the results published in the original paper and discusses the impact of certain differences between the CFGAN framework and the model used in the original evaluation. The absence of random noise and the use of real user profiles as condition vectors leaves the generator prone to learn a degenerate solution in which the output vector is identical to the input vector, therefore, behaving essentially as a simple autoencoder. The work further expands the experimental analysis comparing CFGAN against a selection of simple and well-known properly optimized baselines, observing that CFGAN is not consistently competitive against them despite its high computational cost. To ensure the reproducibility of these analyses, this work describes the experimental methodology and publishes all datasets and source code.