RLNVR: Reinforcement Learning from Non-Verified Real-World Rewards

TL;DR

RLNVR introduces a practical reinforcement learning framework that effectively trains language models using noisy, real-world feedback signals without explicit human verification, improving content quality and training stability.

Contribution

It presents a novel framework combining normalization and reward transfer techniques for training language models from implicit, noisy rewards in real-world settings.

Findings

Significant improvements in content quality.

Enhanced training stability.

Effective use of social media engagement data.

Abstract

This paper introduces RLNVR (Reinforcement Learning from Non-Verified Rewards), a framework for training language models using noisy, real-world feedback signals without requiring explicit human verification. Traditional RLHF requires expensive, verified reward signals that are impractical in many real-world domains. RLNVR addresses this challenge through baseline normalization and semantic similarity-based reward transfer. We demonstrate RLNVR through Walter, a prototype system that optimizes social media content generation using actual engagement data from Bluesky. Our experimental results show significant improvements in content quality and training stability, with comprehensive evaluation planned for future work. Positioning: We present a practical framework that combines RLNVR with GSPO (Group Sequence Policy Optimization) and an optional UED (Unsupervised Environment Design) curriculum to improve stability and diversity under noisy, implicit rewards. To our knowledge, combining GSPO-style normalization with a UED-style curriculum for LLM content generation from implicit social engagement has not been previously documented in this applied setting; we frame this as an applied integration rather than a new algorithm.