Seir\^enes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning

TL;DR

Seir extsuperscript{enes} introduces an adversarial self-play framework that enhances LLM reasoning robustness by co-evolving models to generate and overcome distracting contexts, improving performance across benchmarks.

Contribution

The paper presents a novel self-play RL method where models generate and solve challenging contexts, leading to more resilient reasoning capabilities in LLMs.

Findings

Seir extsuperscript{enes} improves reasoning accuracy by over 7 points on average across seven benchmarks.

Distracting contexts from Seir extsuperscript{enes} models decrease GPT and Gemini accuracy by 4-5 points.

The framework scales effectively from 4B to 30B parameter models.

Abstract

We present Seir\^enes, a self-play RL framework that transforms contextual interference from a failure mode of LLM reasoning into an internal training signal for co-evolving more resilient reasoners. While RL with verifiable rewards has significantly advanced reasoning capabilities, models can still exhibit fragility when encountering non-idealized contexts: scenarios characterized by superfluous information, tangential instructions, or incidental correlations that differ from the clean distributions typical of standard benchmarks. Seir\^enes harnesses this vulnerability through a parameter-shared and adversarial self-play loop. Within this framework, a single model is trained to both construct plausible yet distracting contexts that expose its own reasoning blind spots, and solve problems by discerning the essential task from these perturbations to recover the core underlying logic. By pitting these competing objectives against each other, Seir\^enes compels the model to move beyond superficial pattern matching and anchors its capabilities in robust underlying reasoning. This continuous interaction sustains an informative co-evolutionary curriculum as the model improves. Across seven mathematical reasoning benchmarks and model scales from 4B to 30B, Seir\^enes achieves average gains of +10.2, +9.1, and +7.2 points. Besides, distracting contexts produced by the 4B Seir\^enes model reduce the accuracy of top-tier closed-source models (GPT and Gemini) by roughly 4--5 points, revealing Seir\^enes' general ability to uncover reasoning models' blind spots.