Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
Yifan Sun, Yushan Liang, Zhen Zhang, Xin Liu, Jiaye Teng
TL;DR
This paper develops a theoretical model for understanding how large language models improve through self-training by analyzing the solver-verifier gap, and validates it with experiments across different models and datasets.
Contribution
It introduces a novel theoretical framework for modeling LLM self-improvement dynamics based on solver-verifier gap and extends analysis to external data effects.
Findings
Self-improvement performance is linked to solver-verifier gap.
External data can be used at any stage under limited data regimes.
Theoretical model fits well with experimental results.
Abstract
Self-improvement is a significant techniques within the realm of large language model (LLM), aiming to enhance the LLM performance without relying on external data. Despite its significance, generally how LLM performances evolve during the self-improvement process remains underexplored. In this paper, we theoretically model the training dynamics of self-improvement via the concept of solver-verifier gap. This is inspired by the conjecture that the performance enhancement of self-improvement stems from the gap between LLM's solver capability and verifier capability. Based on the theoretical framework, we further show how to model the entire training trajectory. This framework allows quantifying the capability limit of self-improvement by fitting the theoretical model to the experiment results. We validate the effectiveness of the theoretical framework on various LLMs and datasets. Beyond…
Peer Reviews
Decision·ICLR 2026 Poster
1. First clear formalization of LLM self-improvement dynamics through solver–verifier interactions. 2. The differential-equation approach captures exponential convergence behavior that aligns well with empirical trends. 3. Experiments across datasets and LLMs demonstrate strong fits (R² > 0.9) to the theoretical model, reinforcing its predictive power. 4. The extension to limited external data regimes adds practical insight into data allocation strategies.
1. The model is phenomenological rather than derived from first principles; it lacks a deep mechanistic explanation of why solver-verifier dynamics follow this form. 2. Assumes linear potential energy and time-invariant coefficients (α, β), which may not generalize to all LLM training settings. 3. The connection between theoretical variables (E(t), G(t)) and practical LLM learning signals remains abstract. 4. The paper omits discussion of computational cost, convergence rate sensitivity, and imp
1. The paper studies the important problem of modeling the LLM self-improvement process, which models a very complex system with simple differential equations. 2. The paper provides experiments to verify the validity of the proposed models.
1. I had a difficult time even trying to understand the basic definitons such as solver uncertainty and verifier uncertainty. Are terms like $U_s(t)$ and $U_v(t)$ random variables, since $y_i$ are random variables? If so, how do we even understand these terms and so the gap term? How are they measured in practice? 2. I don't see the necessity of formulating everything as differential equations as the LLM self-improvement update is discrete. 3. The exeperiment setup seems unclear. For example,
1. The paper provides an interesting and effective theoretical exposition of LLM self-improvement, accompanied by a relatively rigorous theoretical proof. 2. It conducts diverse empirical studies on mathematical tasks across multiple models, which enhances the credibility of the results.
1. The relationship between model capability and output uncertainty is not clearly articulated. The connection between a model’s accuracy on different tasks, its output uncertainty, and its underlying “capability” remains ambiguous. Why can output uncertainty serve as a valid indicator of model capability? How is this metric related to output accuracy? Although the authors provide some explanation in the appendix, the paper lacks further theoretical and empirical justification for the appropriat
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Computational and Text Analysis Methods · Natural Language Processing Techniques