MiniRec: Data-Efficient Reinforcement Learning for LLM-based Recommendation

TL;DR

MiniRec introduces a reward-based, trajectory-aligned data selection method for RL-enhanced LLM recommendation systems, significantly reducing training costs while maintaining high performance.

Contribution

It presents MiniRec, a novel data selection framework that aligns sample choice with RL signals and optimization trajectories, improving efficiency in RL-based LLM recommendation.

Findings

Reduces training cost by up to 50%

Maintains recommendation performance with fewer samples

Highlights importance of reward-aligned data selection

Abstract

The integration of reinforcement learning (RL) into large language models (LLMs) has opened new opportunities for recommender systems by eliciting reasoning and improving user preference modeling. However, RL-based LLM recommendation faces significant efficiency challenges, making full-data training costly. Existing data selection methods define sample value based on learnability or representativeness, yet their loss- or gradient-driven or dataset coverage-driven criteria often misalign with RL learning dynamics, resulting in suboptimal performance. To address this, we propose MiniRec, a data selection framework tailored for RL-based LLM recommendation. MiniRec evaluates sample learnability using key RL signals -- rewards -- pruning samples that are too easy (too high reward) or too difficult (consistently low reward). It assesses representativeness by aligning sample gradients with the approximated "ideal" global RL optimization trajectory, selecting samples that mainly drive model updates, and it also enforces diversity to reduce redundancy. Combined with a curriculum learning strategy from easy to hard samples, MiniRec significantly reduces training cost while largely preserving performance. Extensive experiments demonstrate MiniRec's effectiveness, highlighting the importance of reward-aligned, trajectory-informed data selection in RL-based LLM recommendation.