Peer-Predictive Self-Training for Language Model Reasoning

TL;DR

This paper introduces Peer-Predictive Self-Training (PST), a collaborative, label-free fine-tuning method where multiple language models improve through internal feedback without external supervision.

Contribution

PST leverages cross-model aggregated responses and mutual information to enhance self-training, improving reasoning accuracy and reducing the generator-verifier gap without external labels.

Findings

PST improves exact-match accuracy by 2.2 to 4.3 percentage points.

PST reduces the generator-verifier gap by 26 to 40%.

PST requires no external supervision, relying solely on cross-model interactions.

Abstract

Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. We propose Peer-Predictive Self-Training (PST), a label-free fine-tuning framework in which multiple language models improve collaboratively by leveraging a cross-model aggregated response as an internal training signal. Given a prompt question, the models generate responses sequentially; the final aggregated answer, often more reliable than individual responses in practice, serves as an internal target for learning. We measure how informative each intermediate response is about the aggregate using pointwise mutual information (PMI), and use this signal to scale self-training updates. Responses already aligned with the aggregate are updated less, while less informative or misaligned responses are updated more. On mathematical reasoning benchmarks (SimulEq, Math500, and MultiArith), PST improves exact-match accuracy by 2.2 to 4.3 percentage points across Gemma-2-2B, LLaMA-3.2-1B, and Qwen-2.5-1.5B, and reduces the average generator-verifier gap (GV-Gap) by 26 to 40 percent, while requiring no external supervision or teacher-student hierarchy and relying solely on cross-model interactions. These results suggest that cross-model generations and peer-predictive feedback can serve as an effective approach for self-supervised training.