Shielded RecRL: Explanation Generation for Recommender Systems without Ranking Degradation

TL;DR

Shielded RecRL is a reinforcement learning method that generates personalized explanations for recommender systems without compromising their ranking performance, using a two-tower architecture and a composite reward signal.

Contribution

It introduces a novel RL approach with a two-tower architecture and gradient shielding to produce explanations without degrading ranking accuracy.

Findings

22.5% increase in click-through rate on Amazon Books dataset

Maintains recommender's item-ranking performance while improving explanations

Effective balance between explanation quality and ranking stability

Abstract

We introduce Shielded RecRL, a reinforcement learning approach to generate personalized explanations for recommender systems without sacrificing the system's original ranking performance. Unlike prior RLHF-based recommender methods that directly optimize item rankings, our two-tower architecture keeps the recommender's ranking model intact while a language model learns to produce helpful explanations. We design a composite reward signal combining explanation length, content relevance, and coherence, and apply proximal policy optimization (PPO) with a KL-divergence constraint to fine-tune a large language model with only 0.4% of its parameters trainable via LoRA adapters. In experiments on an Amazon Books dataset (approximately 50K interactions in the fantasy and romance genres), Shielded RecRL improved the relative click-through rate (CTR) by 22.5% (1.225x over baseline) while keeping the recommender's item-ranking behavior virtually unchanged. An extensive ablation study confirms that our gradient shielding strategy and reward design effectively balance explanation quality and policy drift. Our results demonstrate that Shielded RecRL enhances user-facing aspects of recommendations through rich, personalized explanations without degrading core recommendation accuracy.