Hierarchical Group-wise Ranking Framework for Recommendation Models

TL;DR

This paper introduces a hierarchical group-wise ranking framework that enhances recommendation model training by generating structured user clusters and applying listwise ranking losses within these groups, improving ranking accuracy and calibration.

Contribution

It proposes a novel hierarchical user clustering method using residual vector quantization combined with listwise ranking losses at each hierarchy level, addressing the limitations of in-batch negative sampling.

Findings

Consistently improves ranking accuracy across experiments.

Enhances model calibration without complex infrastructure.

Provides a scalable solution for industrial recommender systems.

Abstract

In modern recommender systems, CTR/CVR models are increasingly trained with ranking objectives to improve item ranking quality. While this shift aligns training more closely with serving goals, most existing methods rely on in-batch negative sampling, which predominantly surfaces easy negatives. This limits the model's ability to capture fine-grained user preferences and weakens overall ranking performance. To address this, we propose a Hierarchical Group-wise Ranking Framework with two key components. First, we apply residual vector quantization to user embeddings to generate hierarchical user codes that partition users into hierarchical, trie-structured clusters. Second, we apply listwise ranking losses to user-item pairs at each level of the hierarchy, where shallow levels group loosely similar users and deeper levels group highly similar users, reinforcing learning-to-rank signals through progressively harder negatives. Since users with similar preferences and content exposure tend to yield more informative negatives, applying ranking losses within these hierarchical user groups serves as an effective approximation of hard negative mining. Our approach improves ranking performance without requiring complex real-time context collection or retrieval infrastructure. Extensive experiments demonstrate that the proposed framework consistently enhances both model calibration and ranking accuracy, offering a scalable and practical solution for industrial recommender systems.