PIT: A Dynamic Personalized Item Tokenizer for End-to-End Generative Recommendation

TL;DR

PIT introduces a dynamic, end-to-end personalized item tokenizer that co-evolves with generative recommender models, effectively integrating collaborative signals for improved recommendation performance in industrial settings.

Contribution

It proposes a novel co-generative architecture for dynamic item tokenization, enabling joint evolution of index construction and recommendation, addressing stability and scalability challenges.

Findings

Outperforms baseline models on real-world datasets.

Achieves a 0.402% increase in App Stay Time in Kuaishou deployment.

Demonstrates robustness and scalability with a one-to-many beam index.

Abstract

Generative Recommendation has revolutionized recommender systems by reformulating retrieval as a sequence generation task over discrete item identifiers. Despite the progress, existing approaches typically rely on static, decoupled tokenization that ignores collaborative signals. While recent methods attempt to integrate collaborative signals into item identifiers either during index construction or through end-to-end modeling, they encounter significant challenges in real-world production environments. Specifically, the volatility of collaborative signals leads to unstable tokenization, and current end-to-end strategies often devolve into suboptimal two-stage training rather than achieving true co-evolution. To bridge this gap, we propose PIT, a dynamic Personalized Item Tokenizer framework for end-to-end generative recommendation, which employs a co-generative architecture that harmonizes collaborative patterns through collaborative signal alignment and synchronizes item tokenizer with generative recommender via a co-evolution learning. This enables the dynamic, joint, end-to-end evolution of both index construction and recommendation. Furthermore, a one-to-many beam index ensures scalability and robustness, facilitating seamless integration into large-scale industrial deployments. Extensive experiments on real-world datasets demonstrate that PIT consistently outperforms competitive baselines. In a large-scale deployment at Kuaishou, an online A/B test yielded a substantial 0.402% uplift in App Stay Time, validating the framework's effectiveness in dynamic industrial environments.