Listwise Preference Alignment Optimization for Tail Item Recommendation

TL;DR

This paper introduces LPO4Rec, a listwise preference alignment method for tail-item recommendation that improves training efficiency and effectiveness, outperforming baselines and reducing memory usage.

Contribution

It extends the Bradley-Terry model to listwise comparison, derives an optimal policy for efficient training, and introduces strategies to enhance tail-item recommendation performance.

Findings

Outperforms 10 baselines with up to 50% performance gain.

Reduces GPU memory usage by 17.9% compared to DPO.

Effectively improves tail-item recommendation quality.

Abstract

Preference alignment has achieved greater success on Large Language Models (LLMs) and drawn broad interest in recommendation research. Existing preference alignment methods for recommendation either require explicit reward modeling or only support pairwise preference comparison. The former directly increases substantial computational costs, while the latter hinders training efficiency on negative samples. Moreover, no existing effort has explored preference alignment solutions for tail-item recommendation. To bridge the above gaps, we propose LPO4Rec, which extends the Bradley-Terry model from pairwise comparison to listwise comparison, to improve the efficiency of model training. Specifically, we derive a closed form optimal policy to enable more efficient and effective training without explicit reward modeling. We also present an adaptive negative sampling and reweighting strategy to prioritize tail items during optimization and enhance performance in tail-item recommendations. Besides, we theoretically prove that optimizing the listwise preference optimization (LPO) loss is equivalent to maximizing the upper bound of the optimal reward. Our experiments on three public datasets show that our method outperforms 10 baselines by a large margin, achieving up to 50% performance improvement while reducing 17.9% GPU memory usage when compared with direct preference optimization (DPO) in tail-item recommendation. Our code is available at https://github.com/Yuhanleeee/LPO4Rec.