Learning Variable-Length Tokenization for Generative Recommendation

TL;DR

This paper introduces VarLenRec, a novel framework for learning variable-length tokenization in generative recommendation systems, addressing the mismatch in ID length requirements between popular and tail items.

Contribution

It proposes a new variable-length tokenization method with Hyperbolic Residual Quantization and a Soft Length Controller, guided by an information-theoretic framework PIBA, to improve recommendation performance.

Findings

Variable-length IDs outperform fixed-length IDs in recommendation tasks.

Popular items need shorter IDs, tail items benefit from longer IDs.

VarLenRec improves accuracy and efficiency over state-of-the-art methods.

Abstract

Generative recommendation reformulates recommendation as next-token prediction over discrete semantic identifiers (IDs). A fundamental yet unexplored design choice is that existing methods employ fixed-length tokenization for all items, implicitly assuming uniform encoding capacity regardless of item characteristics. Through systematic experiments across four datasets, we discover the Popularity-Length Paradox: popular items achieve optimal performance with short IDs, while tail items require substantially longer codes to capture discriminative semantics. This reveals a critical mismatch where popular items benefit from abundant collaborative signals and require minimal semantic detail, whereas tail items must rely on fine-grained content features due to sparse interaction data. To address this, we propose VarLenRec, a framework for learning variable-length tokenization. We develop Popularity-Weighted Information Budget Allocation (PIBA), an information-theoretic framework proving that optimal ID length should scale as a negative power of popularity. Directly implementing variable-length allocation faces two technical challenges: standard Euclidean residual quantization lacks geometric capacity to support diverse code lengths without distortion, and discrete length decisions are non-differentiable. We address these through Hyperbolic Residual Quantization, which leverages the exponential volume growth of the Poincar\'e ball to naturally stratify encoding capacity, and a Soft Length Controller, which enables differentiable length prediction via continuous layer retention probabilities regularized by PIBA-derived priors. Extensive experiments demonstrate that VarLenRec achieves significant improvements over state-of-the-art methods in recommendation accuracy and training/inference efficiency, revealing the importance of adaptive encoding capacity in generative recommendation.