Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Models

TL;DR

This paper offers the first rigorous theoretical analysis of Self-Rewarding Language Models, explaining their success and providing bounds on their iterative alignment performance, especially highlighting the role of initial model quality and iteration count.

Contribution

It establishes fundamental limits, error bounds, and decay of initial dependence for SRLMs, connecting theoretical guarantees to practical model classes.

Findings

Performance improves at rate ~1/√n with sample size.

Dependence on initial model decays exponentially with iterations.

Theoretical guarantees apply to linear softmax models.

Abstract

Self-Rewarding Language Models (SRLMs) achieve notable success in iteratively improving alignment without external feedback. Yet, despite their striking empirical progress, the core mechanisms driving their capabilities remain unelucidated, leaving a critical gap in theoretical understanding. This paper provides the first rigorous theoretical guarantees for SRLMs. We first establish a lower bound that characterizes the fundamental limits of a single update step, revealing a critical dependence on the quality of the initial model. We then derive finite-sample error bounds for the full iterative paradigm, showing that performance improves at a rate of $\widetilde{\mathcal{O}}\left(1/\sqrt{n}\right)$ with sample size $n$. Crucially, our analysis reveals that the dependence on the initial model decays exponentially with the number of iterations $T$. This provides a formal explanation for why self-rewarding succeeds: it robustly overcomes poor initialization by steering the dynamics toward internal stability and consistency. Finally, we instantiate our theoretical framework for the linear softmax model class, yielding tailored guarantees that connect our high-level insights to practical model architectures.