No Free Lunch: Rethinking Internal Feedback for LLM Reasoning

TL;DR

This paper explores internal, model-derived feedback methods for improving large language model reasoning, offering an alternative to external reward-based reinforcement learning, with mixed results depending on training stage and model tuning.

Contribution

It introduces Reinforcement Learning from Internal Feedback (RLIF), analyzing its theoretical properties and empirical performance, highlighting its potential and limitations in LLM training.

Findings

RLIF can improve early-stage reasoning performance of LLMs.

Performance of RLIF degrades with further training, sometimes below initial levels.

RLIF offers limited benefits for instruction-tuned models.

Abstract

Reinforcement learning has emerged as a powerful paradigm for post-training large language models (LLMs) to improve reasoning. Approaches like Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) have shown strong results, but they require extensive external supervision. We investigate an alternative class of methods, Reinforcement Learning from Internal Feedback (RLIF), which relies solely on intrinsic model-derived signals instead of external rewards. In particular, we leverage unsupervised reward proxies such as token-level entropy, trajectory-level entropy, and self-certainty. Our theoretical analysis shows these internal objectives are partially equivalent, and we empirically evaluate various RLIF strategies on challenging math reasoning benchmarks. Experimental results demonstrate that RLIF can boost the reasoning performance of base LLMs at the beginning phase of the training, matching or surpassing RLVR techniques on these tasks. However, when training progresses, performance degrades even below the model before training. Moreover, we find that RLIF yields little improvement for instruction-tuned models, indicating diminishing returns of intrinsic feedback once an LLM is already instruction-tuned. We further analyze this limitation by mixing model weights and explain the reason of RLIF's training behaviors, providing practical guidelines for integrating internal feedback signals into LLM training. We hope our analysis of internal feedback will inform more principled and effective strategies for LLM post-training.