Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains

TL;DR

This paper investigates how reinforcement learning post-training affects the reasoning process of large language models in math tasks, revealing that it improves local coherence but not necessarily the correctness of solutions.

Contribution

The study introduces a trace coherence measure based on First-Order Logic to analyze the effects of RL post-training on reasoning traces in LLMs.

Findings

RL post-training improves trace coherence, especially on challenging problems.

Enhanced local coherence does not always lead to valid or correct solutions.

Claims of improved reasoning should consider the distinction between coherence and validity.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR)-based post-training of Large Language Models (LLMs) has been shown to improve accuracy on reasoning tasks and continues to attract significant attention. Existing RLVR methods, however, typically treat all tokens uniformly without accounting for token-level advantages. These methods primarily evaluate performance based on final answer correctness or Pass@K accuracy, and yet make claims about RL post-training leading to improved reasoning traces. This motivates our investigation into the effect of RL post-training on intermediate tokens which are not directly incentivized. To study this, we design an experimental setup using the GRPO algorithm with Qwen-2.5-0.5B model on the GSM8K dataset. We introduce trace coherence, a First-Order Logic (FOL)-based measure to capture the consistency of reasoning steps by identifying errors in the traces. We distinguish between trace validity and trace coherence, noting that the former implies logical soundness while the latter measures local coherence via lack of errors. Our results show that RL post-training overall improves trace coherence with the most significant gains on problems where the base model fails but the RL model succeeds. Surprisingly, RL enhances local coherence without necessarily producing valid or correct solutions. This highlights a crucial distinction: improved local coherence in reasoning steps does not guarantee final answer correctness. We argue that claims of improved reasoning via RL must be examined with care, as these may be based on improved trace coherence, which may not translate into fully valid mathematical proofs.