Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders

TL;DR

This paper introduces Windowed Partial AUC (WPAUC) and a RL method called TAWin to improve LLM recommenders by better aligning training objectives with Top-$K$ metrics, validated on real datasets.

Contribution

It provides a theoretical analysis linking RL optimization to partial AUC and proposes WPAUC and TAWin for improved recommendation performance.

Findings

Replacing random negatives with beam-search negatives shifts the objective toward partial AUC.

WPAUC constrains FPR to a window to better match Top-$K$ metrics.

Experiments show state-of-the-art results on four real-world datasets.

Abstract

Reinforcement learning (RL) effectively optimizes Large Language Model (LLM)-based recommenders by contrasting positive and negative items. Empirically, training with beam-search negatives consistently outperforms random negatives, yet the mechanism is not well understood. We address this gap by analyzing the induced optimization objective and show that: (i) Under binary reward feedback, optimizing LLM recommenders with Group Relative Policy Optimization (GRPO) is theoretically equivalent to maximizing the Area Under the ROC Curve (AUC), which is often misaligned with Top-$K$ recommendation; and (ii) Replacing random negatives with beam-search negatives reshapes the objective toward partial AUC, improving alignment with Top-$K$ metrics. Motivated by this perspective, we introduce Windowed Partial AUC (WPAUC), which constrains the false positive rate (FPR) to a window [$\alpha,\alpha+d$] to more directly align with Top-$K$ metrics. We further propose an efficient Threshold-Adjusted Windowed reweighting (TAWin) RL method for its optimization, enabling explicit control over the targeted Top-$K$ performance. Experiments on four real-world datasets validate the theory and deliver consistent state-of-the-art performance.