Generalist Reward Models: Found Inside Large Language Models

TL;DR

This paper reveals that large language models inherently contain a latent reward function within their training, which can be directly used for reinforcement learning to improve alignment without additional reward training.

Contribution

It proves that the endogenous reward in LLMs is theoretically equivalent to offline inverse reinforcement learning, enabling reward elicitation without extra training.

Findings

Endogenous reward can be directly elicited from pre-trained LLMs.

Reinforcement learning with this reward improves model performance.

Method surpasses existing reward models and LLM-as-a-judge approaches.

Abstract

The alignment of Large Language Models (LLMs) is critically dependent on reward models trained on costly human preference data. While recent work explores bypassing this cost with AI feedback, these methods often lack a rigorous theoretical foundation. In this paper, we discover that a powerful generalist reward model is already latently present within any LLM trained via standard next-token prediction. We prove that this endogenous reward is not a heuristic, but is theoretically equivalent to a reward function learned through offline inverse reinforcement learning. This connection allows us to directly elicit a high-quality reward signal from a base (pre-trained or supervised fine-tuned) model without any further training. Critically, we also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model. To our best knowledge, this is the first theoretical proof of the effectiveness of reinforcement learning for LLMs. Our experiments validate this theory, demonstrating that our method not only outperforms existing LLM-as-a-judge approaches but can also surpass explicitly trained reward models. These findings suggest that the reward modeling stage can be replaced by a principled method of eliciting the knowledge already captured during pre-training, heralding a more efficient, powerful, and scalable paradigm for LLMs alignment as well as multi-modal models.