A Worrying Reproducibility Study of Intent-Aware Recommendation Models

TL;DR

This study critically examines recent intent-aware recommendation models, revealing reproducibility issues and that traditional models often outperform neural approaches, highlighting the need for more rigorous research practices.

Contribution

It provides the first comprehensive reproducibility assessment of recent IARS models, exposing methodological flaws and challenging claims of neural model superiority.

Findings

Reproducing five IARS models was often unsuccessful.

Traditional non-neural models outperformed all examined IARS approaches.

The study highlights significant reproducibility and benchmarking issues in IARS research.

Abstract

Lately, we have observed a growing interest in intent-aware recommender systems (IARS). The promise of such systems is that they are capable of generating better recommendations by predicting and considering the underlying motivations and short-term goals of consumers. From a technical perspective, various sophisticated neural models were recently proposed in this emerging and promising area. In the broader context of complex neural recommendation models, a growing number of research works unfortunately indicates that (i) reproducing such works is often difficult and (ii) that the true benefits of such models may be limited in reality, e.g., because the reported improvements were obtained through comparisons with untuned or weak baselines. In this work, we investigate if recent research in IARS is similarly affected by such problems. Specifically, we tried to reproduce five contemporary IARS models that were published in top-level outlets, and we benchmarked them against a number of traditional non-neural recommendation models. In two of the cases, running the provided code with the optimal hyperparameters reported in the paper did not yield the results reported in the paper. Worryingly, we find that all examined IARS approaches are consistently outperformed by at least one traditional model. These findings point to sustained methodological issues and to a pressing need for more rigorous scholarly practices.