Self-Improving AI Agents through Self-Play

TL;DR

This paper develops a formal framework for self-improving AI agents using a flow-based model governed by a Generator-Verifier-Updater operator, establishing conditions for stability and unifying various self-play architectures.

Contribution

It introduces a formal dynamical systems approach to self-improvement in AI, deriving the Variance Inequality for stability and linking multiple architectures under this framework.

Findings

The GVU operator generates a vector field on agent parameters.

The Variance Inequality provides a spectral condition for stability.

Architectures like AlphaZero and GANs satisfy the inequality under certain conditions.

Abstract

We extend the moduli-theoretic framework of psychometric batteries to the domain of dynamical systems. While previous work established the AAI capability score as a static functional on the space of agent representations, this paper formalizes the agent as a flow $\nu_r$ parameterized by computational resource $r$, governed by a recursive Generator-Verifier-Updater (GVU) operator. We prove that this operator generates a vector field on the parameter manifold $\Theta$, and we identify the coefficient of self-improvement $\kappa$ as the Lie derivative of the capability functional along this flow. The central contribution of this work is the derivation of the Variance Inequality, a spectral condition that is sufficient (under mild regularity) for the stability of self-improvement. We show that a sufficient condition for $\kappa > 0$ is that, up to curvature and step-size effects, the combined noise of generation and verification must be small enough. We then apply this formalism to unify the recent literature on Language Self-Play (LSP), Self-Correction, and Synthetic Data bootstrapping. We demonstrate that architectures such as STaR, SPIN, Reflexion, GANs and AlphaZero are specific topological realizations of the GVU operator that satisfy the Variance Inequality through filtration, adversarial discrimination, or grounding in formal systems.