FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

TL;DR

FlexRec introduces a reinforcement learning framework that adapts large language model-based recommenders to diverse, need-specific objectives by addressing reward sparsity and credit assignment issues, leading to significant performance improvements.

Contribution

This work presents FlexRec, a novel RL-based post-training method that enhances LLM recommenders' flexibility and stability in dynamic recommendation scenarios.

Findings

Up to 59% NDCG@5 improvement in need-specific ranking

Up to 109.4% Recall@5 improvement in need-specific ranking

Up to 24.1% Recall@5 improvement in generalization settings

Abstract

Modern recommender systems must adapt to dynamic, need-specific objectives for diverse recommendation scenarios, yet most traditional recommenders are optimized for a single static target and struggle to reconfigure behavior on demand. Recent advances in reinforcement-learning-based post-training have unlocked strong instruction-following and reasoning capabilities in LLMs, suggesting a principled route for aligning them to complex recommendation goals. Motivated by this, we study closed-set autoregressive ranking, where an LLM generates a permutation over a fixed candidate set conditioned on user context and an explicit need instruction. However, applying RL to this setting faces two key obstacles: (i) sequence-level rewards yield coarse credit assignment that fails to provide fine-grained training signals, and (ii) interaction feedback is sparse and noisy, which together lead to inefficient and unstable updates. We propose FlexRec, a post-training RL framework that addresses both issues with (1) a causally grounded item-level reward based on counterfactual swaps within the remaining candidate pool, and (2) critic-guided, uncertainty-aware scaling that explicitly models reward uncertainty and down-weights low-confidence rewards to stabilize learning under sparse supervision. Across diverse recommendation scenarios and objectives, FlexRec achieves substantial gains: it improves NDCG@5 by up to \textbf{59\%} and Recall@5 by up to \textbf{109.4\%} in need-specific ranking, and further achieves up to \textbf{24.1\%} Recall@5 improvement under generalization settings, outperforming strong traditional recommenders and LLM-based baselines.