$\pi$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

TL;DR

This paper introduces $ ext{π}$-Play, a multi-agent self-play framework that uses privileged self-distillation with question construction paths to improve learning efficiency without external data.

Contribution

It proposes leveraging question construction paths as privileged information in self-play, enabling dense supervision and more efficient training of search agents.

Findings

$ ext{π}$-Play surpasses fully supervised agents in performance.

It improves evolutionary efficiency by 2-3 times over traditional self-play.

The method operates without external data or human feedback.

Abstract

Deep search agents have emerged as a promising paradigm for addressing complex information-seeking tasks, but their training remains challenging due to sparse rewards, weak credit assignment, and limited labeled data. Self-play offers a scalable route to reduce data dependence, but conventional self-play optimizes students only through sparse outcome rewards, leading to low learning efficiency. In this work, we observe that self-play naturally produces a question construction path (QCP) during task generation, an intermediate artifact that captures the reverse solution process. This reveals a new source of privileged information for self-distillation: self-play can itself provide high-quality privileged context for the teacher model in a low-cost and scalable manner, without relying on human feedback or curated privileged information. Leveraging this insight, we propose Privileged Information Self-Play ($\pi$-Play), a multi-agent self-evolution framework. In $\pi$-Play, an examiner generates tasks together with their QCPs, and a teacher model leverages QCP as privileged context to densely supervise a student via self-distillation. This design transforms conventional sparse-reward self-play into a dense-feedback self-evolution loop. Extensive experiments show that data-free $\pi$-Play surpasses fully supervised search agents and improves evolutionary efficiency by 2-3$\times$ over conventional self-play.