Beyond the Flat Sequence: Hierarchical and Preference-Aware Generative Recommendations

TL;DR

This paper introduces HPGR, a hierarchical, preference-aware generative recommender system that improves modeling of user behavior and computational efficiency by incorporating structural priors and dynamic attention mechanisms, achieving state-of-the-art results.

Contribution

The paper proposes a novel two-stage framework, HPGR, combining structure-aware pre-training and preference-guided sparse attention for improved recommendation quality and efficiency.

Findings

HPGR outperforms strong baselines on large-scale industrial data.

HPGR achieves significant improvements in recommendation accuracy.

Empirical results confirm the effectiveness of hierarchical and preference-aware modeling.

Abstract

Generative Recommenders (GRs), exemplified by the Hierarchical Sequential Transduction Unit (HSTU), have emerged as a powerful paradigm for modeling long user interaction sequences. However, we observe that their "flat-sequence" assumption overlooks the rich, intrinsic structure of user behavior. This leads to two key limitations: a failure to capture the temporal hierarchy of session-based engagement, and computational inefficiency, as dense attention introduces significant noise that obscures true preference signals within semantically sparse histories, which deteriorates the quality of the learned representations. To this end, we propose a novel framework named HPGR (Hierarchical and Preference-aware Generative Recommender), built upon a two-stage paradigm that injects these crucial structural priors into the model to handle the drawback. Specifically, HPGR comprises two synergistic stages. First, a structure-aware pre-training stage employs a session-based Masked Item Modeling (MIM) objective to learn a hierarchically-informed and semantically rich item representation space. Second, a preference-aware fine-tuning stage leverages these powerful representations to implement a Preference-Guided Sparse Attention mechanism, which dynamically constrains computation to only the most relevant historical items, enhancing both efficiency and signal-to-noise ratio. Empirical experiments on a large-scale proprietary industrial dataset from APPGallery and an online A/B test verify that HPGR achieves state-of-the-art performance over multiple strong baselines, including HSTU and MTGR.