How Well Does Generative Recommendation Generalize?

TL;DR

This paper systematically evaluates how generative recommendation models generalize compared to item ID-based models, revealing their strengths in different instance types and proposing a hybrid approach for improved performance.

Contribution

It introduces a data instance categorization method, analyzes the generalization capabilities at token and item levels, and proposes a hybrid model combining both paradigms for better recommendations.

Findings

GR models excel on instances requiring generalization

Item ID models perform better on memorization tasks

A hybrid approach improves overall recommendation accuracy

Abstract

A widely held hypothesis for why generative recommendation (GR) models outperform conventional item ID-based models is that they generalize better. However, there is few systematic way to verify this hypothesis beyond a superficial comparison of overall performance. To address this gap, we categorize each data instance based on the specific capability required for a correct prediction: either memorization (reusing item transition patterns observed during training) or generalization (composing known patterns to predict unseen item transitions). Extensive experiments show that GR models perform better on instances that require generalization, whereas item ID-based models perform better when memorization is more important. To explain this divergence, we shift the analysis from the item level to the token level and show that what appears to be item-level generalization often reduces to token-level memorization for GR models. Finally, we show that the two paradigms are complementary. We propose a simple memorization-aware indicator that adaptively combines them on a per-instance basis, leading to improved overall recommendation performance.