PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play

TL;DR

PopuLoRA is a population-based self-play framework for reinforcement learning with LLMs, enabling co-evolution of problem difficulty and solution capabilities, leading to improved performance on reasoning and coding benchmarks.

Contribution

It introduces a novel population-based asymmetric self-play method with LoRA weight-space evolution operators for training LLMs in a co-evolutionary setting.

Findings

Population outperforms single-agent baseline on multiple benchmarks.

Co-evolution leads to increasingly complex problems and diverse solutions.

Weakest population member surpasses baseline performance on aggregate.

Abstract

We introduce PopuLoRA, a population-based asymmetric self-play framework for reinforcement learning with verifiable rewards (RLVR) post-training of LLMs. Teachers and students are specialised LoRA adapters on a shared frozen base: teachers propose problems, matched students solve them under a programmatic verifier, and cross-evaluation between sub-populations replaces the self-calibration that limits single-agent self-play. A family of LoRA weight-space evolution operators (mutations and crossovers that produce same-rank population members in seconds) serves as the replacement step of a population-based training loop at 7B scale. We instantiate PopuLoRA on top of Absolute Zero Reasoner and compare it against a per-adapter compute-matched single-agent baseline. Where the single agent self-calibrates to generating easy problems it can reliably solve, the population enters a co-evolutionary arms race: teachers produce increasingly complex problems, student solve rates oscillate, and problem-space coverage keeps expanding throughout training. Despite lower training-time reward, the population mean outperforms the baseline on three code benchmarks (HumanEval+, MBPP+, LiveCodeBench) and seven math benchmarks (AIME 24/25, AMC 23, MATH-500, Minerva, GSM8K, OlympiadBench), and even the weakest member of the population beats the baseline on aggregate.