Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytelling

TL;DR

This paper presents RLCS, a framework combining a novel Generative Reward Model and entropy-based reward shaping to improve reinforcement learning for creative storytelling, achieving higher alignment with human judgments and better story quality.

Contribution

It introduces a systematic approach to reward modeling and training stability in RL for storytelling, including a new reward model and dynamic reward shaping strategies.

Findings

GenRM achieves 68% alignment with human creativity judgments.

RLCS outperforms strong baselines like Gemini-2.5-Pro.

The proposed methods improve story quality and training stability.

Abstract

While Large Language Models (LLMs) can generate fluent text, producing high-quality creative stories remains challenging. Reinforcement Learning (RL) offers a promising solution but faces two critical obstacles: designing reliable reward signals for subjective storytelling quality and mitigating training instability. This paper introduces the Reinforcement Learning for Creative Storytelling (RLCS) framework to systematically address both challenges. First, we develop a Generative Reward Model (GenRM) that provides multi-dimensional analysis and explicit reasoning about story preferences, trained through supervised fine-tuning on demonstrations with reasoning chains distilled from strong teacher models, followed by GRPO-based refinement on expanded preference data. Second, we introduce an entropy-based reward shaping strategy that dynamically prioritizes learning on confident errors and uncertain correct predictions, preventing overfitting on already-mastered patterns. Experiments demonstrate that GenRM achieves 68\% alignment with human creativity judgments, and RLCS significantly outperforms strong baselines including Gemini-2.5-Pro in overall story quality. This work provides a practical pipeline for applying RL to creative domains, effectively navigating the dual challenges of reward modeling and training stability.