RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs

TL;DR

This paper critically examines the structural assumptions underlying RL post-training of large language models, revealing they reduce to supervised fine-tuning and questioning claims of improved reasoning abilities.

Contribution

It identifies key assumptions in modeling LLM training as an MDP that lead to a degenerate formulation, showing RL post-training effectively becomes supervised fine-tuning.

Findings

Filtered Iterative SFT achieves comparable performance to RL-based methods.

Structural assumptions simplify RL to outcome-driven supervised learning.

RL incentivizes longer token sequences, affecting model behavior.

Abstract

Reinforcement learning based post-training of large language models (LLMs) has recently gained attention, particularly following the release of DeepSeek R1, which applied GRPO for fine-tuning. Amid the growing claims around improved reasoning abilities attributed to RL post-training, we critically examine the formulation and assumptions underlying these methods. We start by highlighting popular structural assumptions made in modeling LLM training as an MDP, and show how they lead to a degenerate MDP, that characterizes the problem as a contextual bandit, where RL updates naturally collapse into a form of on-policy variant of outcome-driven supervised learning. The two critical structural assumptions include (1) making the MDP states be just a concatenation of the actions with states becoming the context window and the actions becoming the tokens in LLMs and (2) splitting the reward of a state-action trajectory uniformly across the trajectory. Our comprehensive analysis demonstrates that, due to these simplifying assumptions, GRPO objective reduces to filtered Iterative SFT, an on-policy variant of supervised fine-tuning. Our experiments on benchmarks including GSM8K and Countdown, across a diverse set of model families show that Filtered Iterative SFT, incorporating both positive and negative samples, achieves performance comparable to GRPO-based training. We also show that these structural assumptions indirectly incentivize RL to generate longer sequences of intermediate tokens which in turn feeds into the narrative of "RL incentivizing thinking because it generates longer thinking traces."