When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?

TL;DR

This study systematically compares neural and traditional recommendation models on multiple metrics, revealing that neural models do not always outperform traditional ones and highlighting their strengths in diversity and robustness.

Contribution

It provides a comprehensive, fair evaluation framework and large-scale analysis comparing neural and traditional recommendation models on implicit data.

Findings

Neural models do not dominate traditional models in average HitRate.

Neural models outperform in recommendation diversity.

Neural models show better robustness across user and item subgroups.

Abstract

In recent years, neural models have been repeatedly touted to exhibit state-of-the-art performance in recommendation. Nevertheless, multiple recent studies have revealed that the reported state-of-the-art results of many neural recommendation models cannot be reliably replicated. A primary reason is that existing evaluations are performed under various inconsistent protocols. Correspondingly, these replicability issues make it difficult to understand how much benefit we can actually gain from these neural models. It then becomes clear that a fair and comprehensive performance comparison between traditional and neural models is needed. Motivated by these issues, we perform a large-scale, systematic study to compare recent neural recommendation models against traditional ones in top-n recommendation from implicit data. We propose a set of evaluation strategies for measuring memorization performance, generalization performance, and subgroup-specific performance of recommendation models. We conduct extensive experiments with 13 popular recommendation models (including two neural models and 11 traditional ones as baselines) on nine commonly used datasets. Our experiments demonstrate that even with extensive hyper-parameter searches, neural models do not dominate traditional models in all aspects, e.g., they fare worse in terms of average HitRate. We further find that there are areas where neural models seem to outperform non-neural models, for example, in recommendation diversity and robustness between different subgroups of users and items. Our work illuminates the relative advantages and disadvantages of neural models in recommendation and is therefore an important step towards building better recommender systems.